Ingredient inquiry system, ingredient inquiry method, and ingredient inquiry program

According to an embodiment, an ingredient inquiry system includes a personal-information storing unit, an image input unit, an ingredient discriminating unit, an ingredient specifying unit, and a display. The personal-information storing unit stores, about each of a plurality of individuals, personal information for causing restriction about intake of ingredients used in food and drink. The image input unit inputs an image obtained by photographing the food and drink. The ingredient discriminating unit specifies the food and drink based on the image and discriminates the ingredients used in the food and drink. The ingredient specifying unit collates the personal information corresponding to each of the individuals and the ingredients discriminated by the ingredient discriminating unit and specifies the ingredients corresponding to each of the individuals. The display displays a list of the ingredients corresponding to each of the individuals specified by the ingredient specifying unit.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2020-055888, filed on Mar. 26, 2020, the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an ingredient inquiry system, an ingredient inquiry method, and an ingredient inquiry program.

BACKGROUND

Concerning foods and drinks such as dishes provided in restaurants, in some restaurants, allergy indications or ingredients and the like considered as taboos for religions such as beef and pork are written on menus. However, description targets and standards are different for each of the restaurants. Information that is personally necessary is not always written. We need to ask a store clerk about some information.

Depending on medicines prescribed by doctors and taken, there are ingredients that should not be eaten or drunk such as a grapefruit and caffeine drinks because, for example, the effects of the medicines cannot be obtained. A food and drink providing side cannot deal with such ingredients that should be personally avoided. Foreign visitors speaking different languages can not always communicate with store clerks. It is sometimes difficult to smoothly confirm whether the foreign visitors can take the ingredients used in the food and drink described above.

DETAILED DESCRIPTION

An aspect of embodiments is to provide an ingredient inquiry system, an ingredient inquiry method, and an ingredient inquiry program that make it possible to easily inquire about ingredients used in food and drink for which confirmation is personally necessary.

According to an embodiment, an ingredient inquiry system includes a personal-information storing unit, an image input unit, an ingredient discriminating unit, an ingredient specifying unit, and a display unit. The personal-information storing unit stores, about each of a plurality of individuals, personal information for causing restriction about intake of ingredients used in food and drink. The image input unit inputs an image obtained by photographing the food and drink. The ingredient discriminating unit specifies the food and drink based on the image and discriminates the ingredients used in the food and drink. The ingredient specifying unit collates the personal information corresponding to each of the plurality of individuals and the ingredients discriminated by the ingredient discriminating unit and specifies the ingredients corresponding to each of the plurality of individuals. The display unit displays, as a list, the ingredients corresponding to each of the plurality of individuals specified by the ingredient specifying unit.

An embodiment is explained below with reference to the drawings.

FIG. 1is an overall configuration diagram of a system according to this embodiment. By photographing an image of food and drink such as a dish, the system in this embodiment can inquire about ingredients used in the food and drink. In this embodiment, for example, if personal information for causing restriction about intake of ingredients used in food and drink is set for each of individuals, an inquiry result about a plurality of individuals, for whom different kinds of personal information are respectively set, can be easily acquired by photographing an image of one food or drink. Examples of the restriction about intake of ingredients include allergy ingredients (allergens), medicines being taken, religions, and vegetarian types. Concerning the medicines, there are ingredients that should not be eaten or drunk such as a grapefruit and caffeine drinks because the effects of the medicines cannot be obtained. Concerning the religions, there are ingredients, intake of which is prohibited by doctrines and the like. As the vegetarian types, there are, for example, vegan, dietary vegan, vegetarian, and ovo-lacto vegetarian whose types of restricted ingredients are different. In the personal information, conditions other than the examples described above can be designated.

The system according to this embodiment is configured by an ingredient inquiry system10and electronic devices14connected via a network12such as the Internet.

The ingredient inquiry system10in this embodiment provides an ingredient inquiry service for receiving, from the electronic devices14, images obtained by photographing food and drink and personal information for causing restriction about intake of ingredients used in the food and drink, specifying ingredients corresponding to the personal information (ingredients to be avoided, intake of which should be avoided) included in the food and drink, and causing the electronic devices14to display the ingredients.

The ingredient inquiry system10may be realized by one computer (server) or may be realized by cloud computing in which a plurality of computers connected via the network12cooperate.

The electronic device14is an information processing device used by a user who uses a service provided by the ingredient inquiry system10. The electronic device14is realized by, for example, a smartphone, a personal computer, or a tablet computer. The electronic device14transmits, to the ingredient inquiry system10, an image obtained by photographing food and drink such as a dish and personal information of a person who needs confirmation about whether ingredients to be avoided, intake of which should be avoided, are included in ingredients used in the food and drink. The electronic device14can inquire the ingredient inquiry system10about ingredients targeting a plurality of persons per one dish.

FIG. 2is a block diagram illustrating a main part circuit configuration of a computer that realizes the ingredient inquiry system10. The ingredient inquiry system10includes a processor20, a memory21, a storage device22, and a communication device23.

The processor20controls the units according to an operating system or application programs in order to realize various functions of the ingredient inquiry system10. The processor20is, for example, a CPU.

The memory21includes a nonvolatile memory region and a volatile memory region. The memory21stores the operating system or the application programs in the nonvolatile memory region. The memory21sometimes stores, in the nonvolatile or volatile memory regions, data necessary for the processor20to execute processing for controlling the units. The memory21uses the volatile memory region as a work area where data is rewritten as appropriate by the processor20. The nonvolatile memory region is, for example, a ROM. The volatile memory region is, for example, a RAM.

The application programs stored in the memory21includes, for example, an inquiry system program25and an image analysis AI (Artificial Intelligence) program26. The inquiry system program25is a program for collating a plurality of kinds of personal information received from the electronic devices14and ingredients (an ingredient list) used in food and drink discriminated by processing of the image analysis AI program26and specifying ingredients to be avoided corresponding to each of a plurality of individuals. The image analysis AI program26is a program for performing processing for specifying food and drink based on images obtained by photographing food and drink received from the electronic devices14and processing for discriminating ingredients used in the specified food and drink. For example, the image analysis AI program26specifies food and drink through a visual search for specifying food and drink with an AI technique using a learning model for food and drink recognition prepared in advance.

The storage device22is equivalent to an auxiliary storage part of the computer. The storage device22is configured by, for example, a HDD or an SSD. The storage device22stores, for example, data used by the processor20in performing various kinds of processing or data created by processing in the processor20. The storage device22sometimes stores the application programs.

The storage device22stores, for example, a food and drink database27and an ingredient database28used for processing by the inquiry system program25and the image analysis AI program26.

In the food and drink database27, an ingredient list is registered in which, in association with each of a plurality of foods and drinks specified by the processing by the image analysis AI program26, ingredients used in the food and drink are registered. In the ingredient database28, ingredients to be avoided corresponding to each of a plurality of different kinds of personal information are registered.

The communication device23controls communication with the electronic devices14connected via the network12.

FIG. 3is a diagram illustrating an example of a data configuration of the food and drink database27in this embodiment.

In the food and drink database27, a plurality of different foods and drinks271and ingredient lists272corresponding to the respective foods and drinks are registered. The ingredient lists include data indicating a plurality of ingredients used in the food and drink corresponding to the ingredient lists.

FIG. 4is a diagram illustrating an example of a data configuration of the ingredient database28in this embodiment.

In the ingredient database28, a plurality of different kinds of personal information281designated as inquiry targets from the electronic devices14and ingredients (ingredients to be avoided, intake of which should be avoided)282corresponding to the respective kinds of personal information are registered in association with each other. For example, “ingredient 3” is registered as the ingredient to be avoided about personal information “allergy 2”. As illustrated inFIG. 3, a plurality of ingredients are sometimes registered with respect to one kind of personal information.

FIG. 5is a block diagram illustrating a main part circuit configuration of the electronic device14. The electronic device14includes a processor30, a memory31, a storage device32, an input device33, a display device34, a communication device35, and a photographing device36.

The processor30controls the units according to an operating system or application programs in order to realize various functions of the electronic device14. The processor30is, for example, a CPU.

The memory31includes a nonvolatile memory region and a volatile memory region. The memory31stores the operating system and the application programs in the nonvolatile memory region. The memory31sometimes stores, in the nonvolatile or volatile memory region, data necessary for the processor30to execute processing for controlling the units. The memory31uses the volatile memory region as a work area where data is rewritten as appropriate by the processor30. The nonvolatile memory region is, for example, a ROM. The volatile memory region is, for example, a RAM. Programs stored in the memory31include an image inquiry program37for using an ingredient inquiry service provided by the ingredient inquiry system10. In the memory31, for example, a food and drink image38obtained by photographing food and drink with the photographing device36is stored in order to use the food and drink inquiry service according to the image inquiry program37.

The storage device32is equivalent to an auxiliary storage part of a computer. For example, a HDD or an SDD could be the storage device32. The storage device32saves, for example, data used by the processor30in performing various kinds of processing or data created by processing in the processor30. The storage device32sometimes stores the application programs. In the storage device32, for example, in order to use the ingredient inquiry service, personal information39for causing restriction about intake of ingredients used in food and drink is stored about each of a plurality of individuals.

The communication device35controls communication with the ingredient inquiry system10connected via the network12.

The input device33and the display device34are devices that also function as an input device and a display device of the electronic device14configured as, for example, a touch panel. The input device33detects a touch position on an image displayed by the display device34and outputs information concerning the touch position to the processor30.

The photographing device36is used to photograph an image of food and drink in order to use the ingredient inquiry service.

FIG. 6is a diagram illustrating an example of a data configuration of the personal information39stored in the electronic device14.

As illustrated inFIG. 6, in the personal information39, an individual ID391, an individual name392, and individual restriction information393, which are respectively specific kinds of identification information, are registered in association with a plurality of individuals. The individual restriction information393indicates a reason for causing restriction about intake of ingredients used in food and drink. The individual restriction information393is equivalent to the personal information281registered in the ingredient database28of the ingredient inquiry system10.

The operation of the ingredient inquiry system in this embodiment is explained.

FIG. 7is a block diagram illustrating a flow of processing of the ingredient inquiry system10in this embodiment. InFIG. 7, an inquiry system251is a function realized by executing the inquiry system program25with the processor20. An image analysis AI261is a function realized by executing the image analysis AI program26with the processor20.

FIG. 8is a flowchart illustrating processing of the electronic device14in this embodiment.FIG. 9is a flowchart illustrating processing of the ingredient inquiry system10in this embodiment.

In the electronic device14, the image inquiry program37is installed in order to use the ingredient inquiry service provided by the ingredient inquiry system10. Before the ingredient inquiry service is used, registration processing for the personal information39is executed by the function of the image inquiry program37.

For example, in the registration processing for the personal information39, an individual name is input and individual restriction information, which is a reason for causing restriction about intake of ingredient used in food and drink, is set. Examples of the reason for causing restriction about intake of ingredient include medicines being taken, religions, allergens, and vegetarian types. The electronic device14sets a specific individual ID for a pair of the individual name and the individual restriction information and stores the individual ID as the personal information39.

If the electronic device14is used in a nursery school or the like, personal information about a plurality of nursery school children is registered in advance. Consequently, it is possible to inquire the ingredient inquiry system10collectively about the plurality of nursery school children whether ingredients that the nursery school children cannot eat (that should be paid attention) are included in one dish.

If execution of the image inquiry program37is instructed by operation from the input device33, the processor30of the electronic device14causes, based on the personal information39registered in advance, the display device34to display an individual list for selecting an individual to be set as an inquiry target (Act10inFIG. 8).

FIG. 10is a diagram illustrating an example of an individual list screen displayed on the display device34of the electronic device14. A list display area341in which a plurality of individual names registered in advance are displayed and a decision button342are provided on the individual list screen.

The displayed individual names can be changed by scrolling the list display area341with slide operation or the like on the input device33by the user. In the list display area341, an individual name can be optionally displayed in a selected state by touch operation on the input device33by the user.

If detecting that touch operation is performed in the list display area341, the processor30causes the display device34to display an individual name in the selected state (Act11). A plurality of individual names can be displayed in the selected state (No in Act12). InFIG. 10, “name 1”, “name 3”, and “name 5” are displayed in the selected state by being designated by user operation.

If the selection of the individual names is instructed by user operation on the decision button342(Yes in Act12), the processor30reads out, from the personal information39, individual IDs and individual control information corresponding to the individual names selected in the list display area341and causes the memory31to store the individual IDs and the individual control information as collation target data.

If the selection of the individual names is decided, the processor30starts the photographing device36and shifts to a photographing mode for photographing an image of food and drink set as an inquiry target. If the image of the food and drink is photographed by user operation using an image photographing function provided in the electronic device14, the processor30causes the memory31to store the photographed food and drink image38. The processor30causes the display device34to display the photographed food and drink image38. If the food and drink image38intended by the user is not successfully photographed (Yes in Act14), an image of the food and drink can be repeatedly photographed.

FIG. 11is a diagram illustrating an example of a food and drink image display screen displayed on the display device34of the electronic device14. An image display area343in which a photographed food and drink image is displayed and an ingredient inquiry button344are provided on the food and drink image display screen.

If ingredient image photographing completion and inquiry execution are instructed by user operation on the ingredient inquiry button344(Yes in Act14), the processor30transmits an inquiry request to the ingredient inquiry system10together with the food and drink image38set as an inquiry target and data of personal information (an individual ID and individual restriction information) (Act15) (51inFIG. 7).

If receiving the inquiry request from the electronic device14through the communication device23(Yes in Act20inFIG. 9), the processor20of the ingredient inquiry system10provides data of a food and drink image from the inquiry system251to the image analysis AI261and requests specifying of food and drink requested as a collation target (52in FIG.7).

The image analysis AI261specifies food and drink based on the data of the food and drink image through a visual search for specifying food and drink with an AI technique (Act21). The image analysis AI261searches through the food and drink database27based on the specified food and drink (53inFIG. 7) and acquires an ingredient list corresponding to the food and drink from the food and drink database27(Act22) (54inFIG. 7). The image analysis AI261provides the ingredient list acquired from the ingredient database28to the inquiry system251(55inFIG. 7).

The inquiry system251collates the ingredient list acquired from the image analysis AI261and an ingredient discriminated based on the ingredient database28according to the individual restriction information received from the electronic device14(Act23). Consequently, the inquiry system251discriminates whether ingredients to be avoided, intake of which should be avoided, corresponding to the individual restriction information are included in the food and drink specified based on the food and drink image.

That is, the inquiry system251retrieves the personal information281of the ingredient database28based on individual restriction information for one person (corresponding to one individual ID) (56inFIG. 7). For example, the inquiry system251retrieves “allergy 1” from the personal information281based on individual restriction information “allergy 1”. The inquiry system251acquires the ingredients282registered to correspond to the personal information retrieved from the personal information281(57inFIG. 7). For example, “ingredient 1” and “ingredient 2” can be acquired as ingredients to be avoided for “allergy 1” of the personal information281.

The inquiry system251discriminates whether the ingredients to be avoided corresponding to the individual restriction information acquired from the ingredient database are included in the ingredient list acquired from the image analysis AI261. If the ingredients to be avoided are included in the ingredient list, the inquiry system251stores the relevant ingredients (the ingredients to be avoided) in association with an individual ID corresponding to the individual restriction information set as the processing target.

If receiving individual restriction information of a plurality of individuals (individual IDs) from the electronic device14, in the same manner as explained above, about all kinds of individual restriction information, the inquiry system251discriminates whether ingredients corresponding to individual restriction information for each of the individuals are included in the ingredient list (food and drink) (Acts23and24).

If completing the collation about all the kinds of individual restriction information (Yes in Act24), the inquiry system251transmits an ingredient determination result (an inquiry result) for each of the individuals determined by the collation to the electronic device14and causes the electronic device14to display the ingredient determination result (the inquiry result) (Act25) (58inFIG. 7). That is, the inquiry system251transmits, to the electronic device14, as an inquiry result, a name of the food and drink specified by the image analysis AI261based on the image and data of ingredients to be avoided corresponding to each of a plurality of individual IDs.

If receiving the inquiry result from the ingredient inquiry system10(Yes in Act16inFIG. 8), the processor30of the electronic device14causes the display device34to display the inquiry result.

FIG. 12is a diagram illustrating an example of an inquiry result display screen displayed on the display device34of the electronic device14. The processor30causes, based on data indicating the inquiry result received from the ingredient inquiry system10, the display device34to display a name “food and drink 1” of the food and drink specified based on the food and drink image38and a list of inquiry results. In the list of inquiry results, individual names345of individuals selected as an inquiry target in the list display area341and ingredients to be avoided346set as an ingredient inquiry result are displayed in association with each other.

That is, the processor30discriminates an individual name based on the individual ID included in the inquiry result and causes the display device34to display ingredients associated with the individual name. For example, in the example illustrated inFIG. 12, it is indicated that “ingredient 1” and “ingredient 2”, which are ingredients to be avoided that an individual “name 1” should avoid taking, are included in food and drink “food and drink 1”. It is indicated that an ingredient to be avoided that an individual “name 3” should avoid taking is not included in the food and drink “food and drink 1”.

In this way, the electronic device14photographs an image of food and drink set as an inquiry target, selects an individual who needs confirmation, and transmits an inquiry request to the ingredient inquiry system10. Consequently, it is possible to easily confirm whether ingredients to be avoided, intake of which should be avoided for individuals, are included in food and drink set as a target. That is, simply by photographing food and drink, it is possible to easily inquire whether ingredients, which are allergens, are not included in the food and drink, whether adversely affecting medicines are not included in medicines being taken, and whether ingredients considered taboos for religions are absent. It is unnecessary to designate all ingredients to be avoided themselves for each of individuals in the inquiry. Restricting contents for intake of ingredients (for example, allergy ingredients (allergens), medicines being taken, religions, and vegetarian types) only have to be designated. Therefore, it is easy to confirm ingredients with individuals with whom it is difficult to communicate such as foreign visitors.

The several embodiments are explained above. However, the embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in other various forms. Various omissions, substitutions, and changes can be made without departing from the spirit of the invention. These embodiments and modifications of the embodiments are included in the scope and the gist of the invention and included in the inventions described in claims and the scope of equivalents of the inventions.

The processing described in the embodiment can be provided to various devices while being written in a recording medium such as a magnetic disk (a flexible disk, a hard disk, or the like), an optical disk (a CD-ROM, a DVD, or the like), or a semiconductor memory as a computer-executable program. The processing can also be transmitted by a communication medium and provided to the various devices. The computer reads the program recorded in the recording medium or receives the program via the communication medium and executes the processing by the operation of the computer being controlled by the program.