Artificial intelligence refrigerator

An artificial intelligence refrigerator includes a communication unit, a camera configured to capture an image of a tray including a plurality of grooves for storing food, and a processor configured to control the communication unit and the camera. The processor extracts a marker hidden by the food of a plurality of markers respectively attached to the plurality of grooves from the captured image of the tray, acquires a position identifier of a groove corresponding to the hidden marker and a reception time indicating that the food is received in the groove, transmits the acquired position identifier and the reception time to a server through the communication unit, receives determined color information from the server based on the reception time, and controls a light emitting element provided in the groove to output a color to be displayed on the groove included in the received color information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the National Stage filing under 35 U.S.C. 371 of International Application No. PCT/KR2019/000810, filed on Jan. 21, 2019, the contents of which are hereby incorporated by reference herein its entirety.

TECHNICAL FIELD

The present invention relates to an artificial intelligence refrigerator and, more particularly, to an artificial intelligence refrigerator capable of providing information on a food product received therein.

BACKGROUND ART

A refrigerator is an apparatus capable of keep food fresh for a predetermined period, by cooling a freezing compartment or a refrigerating compartment at a specific temperature while repeating a freezing or refrigerating cycle. In general, the refrigerator includes a main body defining a storage space and a door for opening or closing the storage space. Stored objects such as food may be stored in the storage space and a user may open the door in order to store the stored objects or take out the stored objects.

Recently, with diffusion of refrigerators and development of technology, various requirements for additional functions of the refrigerator are increasing.

In particular, when food such as an egg are stored in a refrigerator through a tray, a function of indicating freshness or an expiration date of the egg has appeared.

Conventionally, a sensor for detecting whether eggs are respectively stored in a plurality of grooves provided in the tray is provided.

However, when the number of grooves configuring the tray increases, the number of sensors increases and thus costs increase.

In addition, since the recommended expiration date cannot be input to the refrigerator, when the eggs are received in the refrigerator, it is impossible to accurately check the freshness of each egg.

PRIOR ART DOCUMENT

Patent Document

DISCLOSURE

Technical Problem

An object of the present invention devised to solve the problem lies in an artificial intelligence refrigerator capable of checking and feeding freshness of food received therein back to a user.

Another object of the present invention devised to solve the problem lies in an artificial intelligence refrigerator capable of more accurately checking freshness of food received therein.

Technical Solution

An artificial intelligence refrigerator according to an embodiment of the present invention can capture images of markers respectively attached to a plurality of grooves configuring a tray, determine whether food is received, and output a visual feedback according to determined freshness based on the reception time of the food and the position identifier of a groove.

An artificial intelligence refrigerator according to an embodiment of the present invention can determine the freshness of the food using a difference between the expiration time of the food received from the mobile terminal of a user and the reception time of the food and output a visual feedback according to the determined freshness.

Advantageous Effects

According to the embodiment of the present invention, as the freshness of the food received in a refrigerator is visually output, a user can immediately check the freshness of the food. Therefore, it is possible to rapidly determine intake and disposal of the food.

According to the embodiment of the present invention, as the expiration date of the food is used to determine the freshness of the food, it is possible to accurately check the freshness of the food.

BEST MODE

Hereinafter, embodiments disclosed in this specification will be described with reference to the accompanying drawings. The accompanying drawings are used to help easily understand the embodiments disclosed in this specification and it should be understood that the technical ideas disclosed in this specification are not limited by the accompanying drawings and include all alterations, equivalents and substitutes within the spirit and scope of the present invention.

FIG.1is a block diagram showing the control configuration of a refrigerator according to an embodiment of the present invention.

Referring toFIG.1, the refrigerator10may include a camera110, a microphone120, a speaker130, a memory140, a communication unit150and a controller160. The components shown inFIG.1are not essential for implementing the refrigerator10and the refrigerator10may include more or fewer components than the components shown inFIG.1.

A display device100may include a touch panel101for receiving touch input from a user and a display102for displaying graphics or text. The touch panel101and the display102may be formed integrally or as separate stacked components.

A processor161included in the controller160may detect types of touch input (e.g., short-touch, long-touch, drag-and-drop, etc.) and acquire the coordinate information of touch input, based on touch input received through the touch panel101.

A display controller162included in the controller160may control operation of the display102based on signals or data received from the processor161. The display102may output graphics or text under control of the display controller162.

The camera110may be disposed in the refrigerator10. When a user opens a door11, a camera controller163included in the controller160may control the camera110to acquire at least one image until the door11is closed. The at least one image acquired by the camera110may be stored in an image database (DB)144of the memory140. In some embodiments, the at least one image acquired by the camera110may be transmitted to a server (not shown) or a terminal through the communication unit150.

In some embodiments, the camera controller163may control the camera110to acquire an image when the opening angle of the door11reaches a predetermined opening angle. The predetermined opening angle may be an angle at which the camera110may acquire an image of the maximum area inside the refrigerator10.

In some embodiments, the camera controller163may control the camera110to acquire the image of the inside of the refrigerator10based on a specific control command received through an input unit such as the touch panel101or the microphone120.

Meanwhile, an image processor164of the controller160may extract a food image from the image acquired by the camera110through various image processing schemes. Alternatively, the image processor164may detect an image change caused by putting or taking food in or out of the refrigerator, from images acquired before and after opening the door11. The processor161or the image processor164may detect food put in or taken out of the refrigerator based on the detected image change.

The microphone120may acquire voice from the user and provide the acquired voice to the processor161. For example, the voice may be related to a function to be provided to the user with respect to particular food.

The voice may include at least one partial voice. The partial voice may be identification information (e.g., a food name) of the particular food or a command related to the function. That is, the voice may include at least one of the identification information of the particular food or the command.

The speaker130may output information related to operation of the refrigerator10in the form of sound or voice. In some embodiments, the refrigerator10may output information related to operation of the refrigerator10through the display device100.

The memory140may store control data or an algorithm for operation of the components included in the refrigerator10. Meanwhile, the memory140may include a food management dB142for storing a variety of information related to foods stored in the refrigerator10and an image DB144for storing the image acquired by the camera110. In some embodiments, the food management DB142and the image DB144may be located in a server connected to the refrigerator10through the communication unit150.

The communication unit150may include at least one communication module for connecting the refrigerator10with a terminal of a user or a server. For example, the communication unit150may include a wireless Internet module such as Wi-Fi and a short-range wireless communication module such as Bluetooth.

The controller160may control overall operation of the refrigerator10. The controller160may control at least one component included in the refrigerator10, in order to provide the user with the functions supported by the refrigerator10.

The controller160may include at least one CPU, AP (application processor), controller, a microcomputer, an integrated circuit, etc. For example, as shown inFIG.1, the controller160may include the processor161, the display controller162, the camera controller163, the image processor164, and a voice recognition module165, without being limited thereto.

Meanwhile, the controller160or the processor161according to the embodiment of the present invention may input the voice acquired by the microphone230to the artificial intelligence based voice recognition module165, thereby recognizing the voice. The voice recognition module165may recognize the at least one partial voice included in the voice and output a result of recognition including the recognized at least one partial voice.

As described above, the partial voice may be the identification information (e.g., the food name) of the particular food or the command related to the function. That is, the voice may include at least one of the identification information of the particular food or the command.

For example, the voice recognition module165may include a voice recognizer having an artificial neural network (ANN) learned through machine learning, thereby recognizing the received voice and outputting the result of recognition including the identification information or characteristic information.

The processor161may perform a function related to food management based on the result of recognition of the voice recognition module165and touch input acquired through the touch panel101.

FIG.2is a ladder diagram illustrating a method of operating an artificial intelligence system according to an embodiment of the present invention.

Referring toFIG.2, the camera110of the refrigerator10captures an image of a tray (S201).

In one embodiment, the camera controller163may periodically capture the image of the tray.

The tray may include a plurality of grooves, in which food products are respectively stored.

The food products may be eggs. This is merely an example and any food products capable of being stored may be used.

The camera110may be disposed in an upper space spaced apart from the tray.

The arrangement structure of the camera110and the tray will be described with reference toFIGS.3to5.

FIGS.3to5are views illustrating the arrangement structure between a camera and a tray according to an embodiment of the present invention.

Referring toFIG.3, the tray300may be an element which may be accommodated in the refrigerator10.

The tray300may include a plurality of grooves310to340.

An egg may be stored in each of the plurality of grooves310to340. For example, a first egg may be stored in a first groove310and a second egg32may be stored in a second groove320.

Each of the plurality of grooves310to340may be an elliptical storage container capable of storing an egg.

The first egg31may be stored in the first groove310and the second egg32may be stored in the second groove320.

Each groove may be attached with a marker for identifying the position of each groove.

For example, the first groove310may be provided with a first marker311for identifying the position of the first groove310, the second groove320may be provided with a second marker321for identifying the position of the second groove320, the third groove330may be provided with a third marker331for identifying the position of the third groove330, and the fourth groove340may be provided with the fourth marker341for identifying the position of the fourth groove340.

In one embodiment, the marker may be any one of a QR code or a barcode. However, this is merely an example.

The camera110may be disposed at the upper side spaced apart from the tray300.

The camera110may periodically capture the image of the tray300. The camera110may periodically capture the image of the tray300, in order to determine whether one or more of the plurality of markers respectively attached to the plurality of grooves included in the tray300is hidden.

The capturing period of the camera110may be 5 minutes, but this is merely an example. The camera may capture the image of the tray in real time.

FIG.4is a plan view of the tray300.

FIG.4shows the image of the tray captured by the camera110.

Each of the plurality of grooves of the tray300may be attached with one or more light emitting diode (LED).

That is, the first groove310may be attached with a first LED410and the second groove320may be attached with a second LED420.

Meanwhile, when the first egg31is stored in the first groove310, the first marker311shown inFIG.3may be hidden.

Similarly, when the second egg32is stored in the second groove320, the second marker shown inFIG.3may be hidden.

FIG.5is a side view showing arrangement of the camera110and the tray300disposed in the refrigerator10.

The processor161of the refrigerator10determines whether a hidden marker is present in the plurality of markers respectively attached to the plurality of grooves included in the captured image of the tray (S203).

The processor161may determine whether a marker hidden by a food product is present in the plurality of markers included in the image of the tray.

In one embodiment, the processor161may store a plurality of position identifiers for respectively identifying the plurality of markers disposed in the plurality of grooves in the memory140.

The processor161may compare the plurality of prestored position identifiers with position identifiers acquired through the image of the tray.

When any one position identifier is not detected through the image of the tray according to the result of comparison, the processor161may determine that the marker of the position identifier is hidden.

That is, when any one position identifier is not detected through the image of the tray, the processor161may determine that a food product is stored in a groove corresponding to the position identifier and the marker is hidden.

This will be described in greater detail with reference toFIG.4.

InFIG.4, assume that the marker is a QR code.

The QR code attached to each groove includes a position identifier for identifying the position thereof.

The processor161may read the QR codes included in the image of the tray and detect position identifiers.

For example, assume that the tray300is provided with only four grooves310to340and eggs are stored in the first groove310and the second groove320.

In addition, assume that the memory140stores the position identifiers of the QR codes respectively attached to the four grooves310to340.

The processor161may read the QR code of the third groove330and the QR code of the fourth groove340and acquire the position identifier of the third groove330and the position identifier of the fourth groove340.

Since the position identifier of the first groove310and the position identifier of the second groove320are not acquired, the processor161may determine that the eggs are stored in the first groove310and the second groove320.

The processor161of the refrigerator10acquires the position identifier for identifying the position of the groove corresponding to the hidden marker and a reception time when the food product is received (S205).

In one embodiment, the reception time of the food product may indicate a time when the marker of the groove is hidden.

The processor161may acquire the time when the marker of the groove is hidden, from the time when the camera110captures the image of the tray.

When the marker hidden by the food is extracted, the processor161may acquire a time when the camera110captures the image of the tray as the reception time.

The processor161of the refrigerator10transmits the position identifier of the groove corresponding to the hidden marker and the reception time to the server200through the communication unit150(S207).

According to another embodiment, the processor161may transmit, to the server20, the position identifier of the groove corresponding to an unhidden marker and a reception time.

The server20may store the plurality of position identifiers respectively corresponding to the plurality of grooves.

The server20may extract the position identifier of the groove corresponding to the hidden marker through the position identifier of the groove corresponding to the unhidden marker received from the refrigerator10.

The server20calculates the freshness of the food product based on the reception time received from the refrigerator10(S209).

In one embodiment, the server20may calculate the freshness of the food product expressed by a numerical value based on the reception time received from the refrigerator10.

The server20may calculate a difference between a current time and the reception time.

The server20may calculate freshness which decreases as the difference between the current time and the reception time increases, and calculate freshness which increases as the difference between the current time and the reception time decreases.

For example, the freshness may be 90 when the difference between the current time and the reception time is one day, and the freshness may be 70 when the difference between the current time and the reception time is three days.

In another embodiment, the server20may calculate freshness based on the reception time received from the refrigerator10and the expiration date of the food product acquired from the mobile terminal (not shown) of the user.

The user may execute a food product management application installed in the mobile terminal (not shown) and input the expiration date of the food product on the execution screen of the executed food product management application.

The expiration date of the food product may be transmitted to the server20.

A process of acquiring freshness based on the expiration data input by the user and the reception time will be described below.

The server20determines a color to be displayed on the groove using the calculated freshness (S211).

The server20stores correspondence between each of a plurality of freshnesses and each of a plurality of colors in the form of a table.

The server20may extract the color corresponding to the freshness and determine the extracted color as the color to be displayed on the groove.

FIG.6is a table showing an example of determining a color to be displayed on a groove, in which a food product is stored, based on freshness of the food product according to an embodiment of the present invention.

In the table600ofFIG.6, assume that the current time is 01/11 15:00 and the expiration date of the food product is 10 days.

The expiration date of the food product may be received from the mobile terminal of the user.

The server20may receive the position identifier of the groove having the hidden marker and the reception time when the food product is received in the groove, from the refrigerator10.

The server20may calculate a retention period indicating the difference between the current time and the reception time.

The server20may calculate the freshness of the food product using the expiration date and the retention period.

The server20may calculate the freshness of the food product through Equation 1 below.
Freshness=(expiration date−retention period)×10  [Equation 1]

For example, the freshness of the food product received in the first groove310having a position identifier value of 1 may be 10 through (10−9)×10.

The freshness of the food product received in the third groove330having a position identifier value of 3 may be 50 through (10−5)×10.

The freshness of the food product received in the fourth groove340having a position identifier value of 4 may be 70 through (10−3)×10.

The freshness of the food product received in the seventh groove (not shown) having a position identifier value of 7 may be 90 through (10−1)×10.

Meanwhile, the server20may store a color corresponding to a freshness range.

For example, the color to be displayed on the groove may be green when the freshness is between 90 and 100, may be yellow when the freshness is between 70 and 80, may be orange when the freshness is between 50 and 60, and may be red when the freshness is between 10 and 20.

The server20may determine the color according to the calculated freshness of the food product and transmit information on the determined color to the refrigerator10through the communication unit (not shown).

The server20transmits, to the refrigerator10, color information including the color to be displayed on the groove (S213).

The server20may transmit the color information of each of the plurality of grooves corresponding to the hidden markers to the refrigerator10.

The processor161of the refrigerator10controls the LED provided in the groove to output the color, using the color information received from the server20(S215).

In one embodiment, the color information may include the position identifier of the groove and the color determined in correspondence with the position identifier of the groove.

For example, when the color information indicating that the color to be displayed on the first groove310is green is received from the server20, the processor161may control operation of the first LED410provided in the first groove310to output green light.

To this end, the tray300and the processor161may perform communication through short-range wireless communication module.

That is, the processor161may transmit a color display command for displaying a particular color on a specific groove to the tray300through the short-range wireless communication standard.

In another example, when color information indicating that the color to be displayed on the second groove320is red is received from the server20, the processor161may control operation of the second LED420provided in the second groove320to output red light.

The user can intuitively determine the freshness of the food product stored in the groove, by outputting the color of the LED.

When the food product is an egg, the user may visually check information indicating eggs to be eaten early, eggs incapable of being eaten and eggs which may be eaten later, among the eggs stored in the refrigerator10.

Meanwhile, according to another embodiment of the present invention, the refrigerator10may display the numerical value of the freshness of the food product stored in each groove through the display102.

In another example, the refrigerator10may display the freshness of the food product stored in each groove through the display102along with the image of the egg stored in the groove.

FIG.7is a flowchart illustrating a method of operating a refrigerator according to an embodiment of the present invention.

In particular,FIG.7is a view illustrating an example in which the refrigerator10autonomously calculates the freshness of the food product and directly determines the color to be displayed on the groove in which the food product is stored, without interworking with the server20as in the embodiment ofFIG.2.

InFIG.7, a portion overlapping with the description ofFIG.2is replaced with the description ofFIG.2.

The camera110of the refrigerator10captures an image of a tray (S701).

The processor161of the refrigerator10determines whether a hidden marker is present in a plurality of markers respectively attached to a plurality of grooves included in the captured image of the tray (S703).

The processor161of the refrigerator10acquires the position identifier for identifying the position of the groove corresponding to the hidden marker and a reception time indicating a time when the food product is received (S705).

The processor161of the refrigerator10calculates the freshness of the food product based on the reception time (S707).

The processor161may calculate a difference between a current time and the reception time.

The processor161may calculate freshness which decreases as the difference between the current time and the reception time increases, and calculate freshness which increases as the difference between the current time and the reception time decreases.

For example, the freshness may be 90 when the difference between the current time and the reception time is one day, and the freshness may be 70 when the difference between the current time and the reception time is three days.

In another embodiment, the processor161may calculate freshness based on the reception time and the expiration date of the food product acquired from the mobile terminal (not shown) of the user.

The user may execute a food product management application installed in the mobile terminal (not shown) and input the expiration date of the food product on the execution screen of the executed food product management application.

The input expiration date of the food product may be transmitted to the refrigerator10.

In another example, the user may input the expiration date of the food product through the touch panel101of the refrigerator10.

The processor161of the refrigerator10determines a color to be displayed on the groove using the calculated freshness (S709).

The refrigerator10stores correspondence between each of a plurality of freshnesses and each of a plurality of colors in the memory140in the form of a table.

The processor161may extract the color corresponding to the freshness and determine the extracted color as the color to be displayed on the groove, using the table stored in the memory140.

The processor161may calculate the freshness of the food product using the embodiment ofFIG.6.

The processor161of the refrigerator10controls the LED provided in the groove to output the color using the determined color information (S711).

According to the embodiment ofFIG.7, the refrigerator10can autonomously provide the user with the freshness of the food product stored in the tray300, without interworking with the server20.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are to be construed as illustrative and not restrictive, and the scope of the technical idea of the present invention is not limited by these embodiments.

The scope of the present invention should be construed according to the following claims, and all technical ideas within equivalency range of the appended claims should be construed as being included in the scope of the present invention.