Patent Description:
A refrigerator is an electronic apparatus (or home appliance) which is capable of keeping food which is edible and drinkable cold and frozen through a refrigeration cycle using a refrigerant. A refrigerator may store not only food but also medicine, alcoholic liquor or cosmetics.

The advancement of technology has enabled a refrigerator to be used as a storage but also to provide a variety of services through data transmission/reception or an installed (or downloadable) application.

Conventional refrigerators and corresponding control methods are disclosed in <CIT>, <CIT>, <CIT>, <CIT> and <CIT>.

The present invention is uniquely defined by the subject-matter indicated in the appended set of claims.

According to an aspect of the present disclosure, the refrigerator may include a user approach detector configured to detect an approach of a user with respect to the refrigerator. The user approach detector may include at least one of a microphone, the first camera and a proximity sensor.

The food management method may further include distinguishably displaying a location at which the food is stored in a storage within the refrigerator according to the food information.

The location at which the food is stored may be distinguishably displayed on an inside of the storage photographed by a second camera.

According to an example embodiment, there is provided a refrigerator, comprising: a main body having a storage; a storage configured to store a first comparison food image; a door configured to be hinged to the main body to rotate on a front side of one side of the storage and have a display on a front surface of the door; a camera configured to be positioned on a front surface of the door and photograph food to be recognized; and a controller configured to control the storage, the display and the camera. The controller may preferentially recognize a food image photographed by the camera according to a food recognition search order by using the first comparison food image of the storage and control a food name corresponding to the photographed food image to be displayed on the display.

According to another example embodiment, there is provided a main body having a storage, a first door configured to hinged to the main body to rotate on a front side of one side of the storage and have a display on a front surface of the first door; a first camera configured to photograph food to be recognized, the first camera being positioned on a front surface of the first door, a second door configured to be hinged to the main body to rotate on a front side of the other side of the storage; and a controller configured to control the display and the first camera. The controller controls a food name corresponding to the food image on the display by using a food image photographed by the first camera.

According to another example embodiment, there is provided a main body having a storage, a storage configured to store a comparison food image, a door configured to hinged to the main body to rotate on a front side of one side of the storage and have a display on a front side of the door, a camera configured photograph food to be recognized, the camera being positioned on a front surface of the door; and a controller configured to control the display and the camera. The controller recognizes a food image photographed by the camera by using a pre-stored comparison food image according to a food recognition search order with respect to a plurality of storages and controls a food name corresponding to the photographed food image to be displayed on the display. The plurality of storages include the storage, a storage of the server and a storage of a store server.

According to an example embodiment, the refrigerator may further include a communication interface. The controller may recognize a food image photographed by the camera by using a second comparison food image stored in a storage of an external server connected via the communication interface according to the food recognition search order and control a food name corresponding to the photographed food image to be displayed on the display.

According to an example embodiment, the controller may recognize a food image photographed by the camera by using a third comparison food image stored in a storage of a store server connected via the communication interface according to the food recognition search order and control a food name corresponding to the photographed food image to be displayed on the display.

According to an example embodiment, the controller <NUM> may recognize a food image photographed by the camera by using a search engine via the communication interface according to the food recognition search order and control a food name corresponding to the photographed food image to be displayed on the display.

A refrigerator in which a display and a first camera adjacent to the display are positioned on a front surface of a door may be provided.

A refrigerator in which a display and a first camera positioned to be inclined from a front surface of a door at a set angle on a front surface of the door may be provided.

A refrigerator which is capable of recognizing a food image photographed by a first camera of the front surface of the door and a food management method of the refrigerator may be provided.

A refrigerator which is capable of recognizing a food image photographed by a first camera of the front surface of the door through machine learning and a food management method of the refrigerator may be provided.

Through a first camera of a front surface of a door and machine learning, the user may be provided with a refrigerator in which food is easily registrable and a food management method of the refrigerator.

A refrigerator which is capable of recognizing a food image photographed by a first camera of the front surface of the door through deep learning and a food management method of the refrigerator may be provided.

Through a first camera of a front surface of a door and deep learning, the user may be provided with a refrigerator in which food is easily registrable and a food management method of the refrigerator.

A refrigerator which is capable of recognizing a food image photographed by a first camera of the front surface of the door by using a feature point of image and a food management method of the refrigerator may be provided.

A refrigerator which is capable of recognizing a food image photographed by a first camera of the front surface of the door by using a feature point of image and/or a feature amount of image and a food management method of the refrigerator may be provided.

Through a first camera of a front surface of a door and a second camera which photographs a storage, a refrigerator managing food and a food management method of the food may be provided.

Through a first camera of a front surface of a door and a second camera which photographs a storage, a refrigerator managing food storage period and a food management method of the food may be provided.

According to the various example embodiments, through a first camera of a front surface of the door, a second camera which photographs a storage and machine learning, the user may be provided with a refrigerator in which food is easily registrable and a food management method of the refrigerator may be provided.

It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation; the term "or," is inclusive, meaning and/or; the phrases "associated with" and "associated therewith," as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term "controller" means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Exemplary embodiments of the present disclosure will now be described in greater detail with reference to the accompanying drawings. In the following description, same drawing reference numerals are used for the same elements even in different drawings.

Terminology including an ordinal number such as "first," "second," and so on may be used to describe a variety of constituent elements, but the elements are not limited by the terminology. In embodiments of the present disclosure, relational terms such as first and second, and the like, may be used to distinguish one entity from another entity, without necessarily implying any actual relationship or order between such entities. For example, a first element may be named a second element without departing from the scope of right of various example embodiments, and similarly, a second element may be named a first element.

According to an example embodiment, the application refers to an operation system (OS) for computers or software executed on a mobile OS to be used by a user. For example, the application may include a web browser, a camera application, a mobile payment application (or, electronic payment application or payment application), a photo album application, a word processor, a spread sheet, a contacts application, a calendar application, a memo application, an alarm application, a social network system (SNS), a call application, a game store, a game application, a chatting application, a map application, a music player or a video player.

According to an example embodiment, an application may refer to a refrigerator or software which is executed in an external device (e.g., wearable device, server, and the like) connected to the refrigerator via wire or wirelessly. According to an example embodiment, an application may refer to software which is executed in a refrigerator in response to a user input that is received.

A content may be executed or displayed through an application which is executed correspondingly. For example, a content may include a video file or audio file which is executed in a video player, which is one of the application, a video file or an audio file, a game file which is executed in a game application, a music file which is reproduced in a music player, a photo file which is displayed in a photo album application, a web page file which is displayed in a web browser, or payment information (e.g., mobile card number, payment amount, product name, service name or store name) which is displayed on an electronic payment application. In addition, a content may include a call screen (e.g., including caller ID, caller name, call start time, or caller video (or caller image) by a video call) which is displayed in a call application.

A content may include a screen of an application (or widget) being executed and a user interface which configures the application screen. In addition, a content may include one or a plurality of contents as well.

The widget refers to a mini application which is one of the GUIs for facilitating an interaction between a user and an application or between the user and an operating system (OS). For example, the widget may include a weather widget, a calculator widget, a clock widget, etc..

In an example embodiment, "user input" may be a term which includes, for example, selection of a button (or key) by a user, depress (or click) of a button (or key) by a user, touch of a soft button (or soft key) by a user, a touch (including non-contact such as hovering) received (or detected) in a touch screen by a user, a touch gesture (including non-contact such as hovering, a user voice, a user presence (e.g., a user appears within a recognition range of a camera) received (or detected) in a touch screen by a user, or a user motion. In addition, 'selection of a button (or key)' may be used to refer to a press (or click) of a button (or key) or a touch of a soft button (or soft key).

Terms used in the present specification are used only in order to describe specific exemplary embodiments rather than limiting the present disclosure. Throughout this specification, it will be understood that the term "comprise" and variations thereof, such as "comprising" and "comprises", specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof, described in the specification, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Like reference numerals in the drawings denote members performing substantially the same function.

1A illustrates a perspective view of a refrigerator having a front camera and a display, according to an embodiment of the present disclosure.

<FIG> illustrates a front view of a refrigerator of which all doors are opened, according to an embodiment of the present disclosure;.

Referring to FIGS. 1A and <FIG>, a refrigerator <NUM> includes a main body <NUM>, a storage <NUM>, <NUM>, <NUM>, a door <NUM>, <NUM>, <NUM>, <NUM>, and a hinge <NUM> which connects each of the doors <NUM>, <NUM>, <NUM>, <NUM> with the main body <NUM>. A display (or touch screen, <NUM>) which displays a content may be positioned on at least one (e.g., at least one of a right door <NUM> and a left door <NUM>) of a plurality of doors.

A camera (or front camera, <NUM>) may be positioned on at least one (e.g., at least one of a right door <NUM> and a left door <NUM>) of a plurality of doors. A proximity sensor <NUM> may be positioned to be adjacent (e.g., within a radius of <NUM>) to the front camera <NUM>. In addition, a microphone <NUM> may be positioned on at least one (e.g., at least one of a right door <NUM> and a left door <NUM>) of a plurality of doors.

The refrigerator <NUM> may be divided into types according to a shape of a storage and a door. A top mounted freezer (TMP) type refrigerator includes a freezer which is formed on an upper surface of the storage horizontally partitioned by a partition and a cold room which is formed on the lower side of the storage. A bottom mounted freezer (BMF) type refrigerator includes a refrigerator which is formed on an upper side of a storage horizontally partitioned by a partition and a freezer which is formed on the lower side of the storage.

A side by side (SBS) type refrigerator includes a freezer which is formed on one side of the storage vertically partitioned by a partition and a cold room which is formed on the other side of the storage. A French door refrigerator (FDR) refrigerator includes a cold room which is formed on an upper side of the storage horizontally partitioned by a partition and a freezer which is formed on a lower side of the storage. A cold room on the upper side may be opened and closed by a pair of doors. In the FDR refrigerator, both the upper cold room and the lower cold room may be opened and closed by a pair of doors.

The main body <NUM> includes an inner case (not illustrated) which forms the storage <NUM> to <NUM>, an outer case (not illustrated) which forms an exterior of the refrigerator, and an insulator (not illustrated) which maintains a temperature difference between the inner case (not illustrated) and the outer case (not illustrated). An insulation material (not illustrated) may prevent a cold inner air from leaking outside, and prevent the outside warm air from entering the storage <NUM> to <NUM>.

The main body <NUM> includes a cool air supply unit (not illustrated) which supplies cool air to the storage <NUM> to <NUM>. The cool air supply unit (not illustrated) may include a compressor <NUM> (see <FIG>) which compresses a refrigerant, a condenser (not illustrated), an expansion valve (not illustrated), an evaporator (not illustrated), and a pipe (not illustrated).

The storage <NUM> to <NUM> is divided by a partition <NUM>. The storage <NUM> to <NUM> is divided into a lower freezer storage room <NUM> and <NUM> (hereinafter referred to as "freezer") and a cold storage room <NUM> (hereinafter referred to as "cold room" above the freezer room <NUM> and <NUM>. The storage <NUM> may be set to a temperature of over zero (e.g., between <NUM> to <NUM>) or below zero (e.g., between -<NUM> to -<NUM>) and store water, beverages, food ingredient, and cold or frozen food. Water or beverages may be stored in a beverage container.

The cold room <NUM> from among the storage <NUM> to <NUM> which is divided by a partition <NUM> may include one or more shelves <NUM> and one or more storage boxes <NUM>.

The cold room <NUM> is attached to one side (e.g., left side) of the storage <NUM> and a second door <NUM> which is adjacent to the first door <NUM> and positioned on the other side (e.g., right side). The first door <NUM> and/or the second door <NUM> may be rotated at an angle (e.g., <NUM>° or less) set by each hinge <NUM> and open or close (e.g., attach or detach) a front surface of the storage <NUM>. The first door <NUM> may be rotated the opposite way to a rotation direction of the second door <NUM> and open or close the storage <NUM>. The positions of the first door <NUM> and the second door <NUM> may be mutually changed.

The first door <NUM> is rotated at an angle (e.g., <NUM>°or less) set by a hinge <NUM> and open or close a portion of the front surface of the storage <NUM> (e.g., between <NUM>% to <NUM>% of the front surface of the storage <NUM>).

A dispenser (not illustrated) which provides water, ice or sparkling water to the front surface (e.g., +y-axis direction) of the first door <NUM> and/or a grabbable handle <NUM>.

The second door <NUM> is rotated at an angle (e.g., <NUM>°or less) set by a hinge <NUM> and open or close a portion of the front surface of the storage <NUM> (e.g., between <NUM>% to <NUM>% of the front surface of the storage <NUM>). The second door <NUM> may include a grabbable handle <NUM>. The handle <NUM> of the first door <NUM> and the handle <NUM> of the second door <NUM> are positioned to be spaced apart in the left and right directions based on a center area of the storage <NUM>.

A function and stored setting of the refrigerator <NUM> may be displayed on the front surface (e.g., +y-axis direction) of the second door <NUM>. A user input (e.g., touch or selection of a button <NUM>) may be received in the front surface of the second door <NUM>. A display (or a touch screen <NUM>) which is capable of displaying (or executing) a screen of an application (including a widget) may be positioned at the front surface of the second door <NUM>. The button <NUM> may be included in the touch screen <NUM> or may be positioned separately from the touch screen <NUM>. The button <NUM> may be a button which is displayed on a touch screen, a touch button or a physical button.

A vegetable compartment 11d1 and 11d2 is positioned below the storage <NUM>. The vegetable compartment 11d1, 11d2 may be withdrawn (e.g., slid or rolled) toward the front side (e.g., y-axis direction).

The storage <NUM> may have a door <NUM> on one side. The storage <NUM> may have a door <NUM> on one side. The storage <NUM> and <NUM> may be combined into one storage (e.g., like the storage <NUM>). One storage (not illustrated) may have a door (not illustrated) on each of the left side and the right side like the storage <NUM>. In addition, a refrigerator may have a drawer (not illustrated).

<FIG> illustrates a block diagram of a refrigerator having a front camera and a display, according to an embodiment of the present disclosure.

Referring to <FIG>, a refrigerator <NUM> may be functionally connected with an external apparatus <NUM> by using a communication interface <NUM>. The external apparatus <NUM> may include a server <NUM>, an external apparatus <NUM> including a smartphone, a home appliance apparatus <NUM> including an air conditioner, a washing machine and the like, or a display apparatus <NUM> including a TV.

The refrigerator <NUM> may transmit operation information corresponding to an operation (e.g., temperature control of each storage) of the refrigerator and state information corresponding to a state (e.g., normal, abnormal, and the like) of the refrigerator <NUM> to the external apparatus <NUM> via the communication interface <NUM>, or may receive control information (e.g., a control command corresponding to quick freezing of the refrigerator <NUM>) from an outside source.

The refrigerator <NUM> may include a controller <NUM>, a communication interface <NUM>, a microphone <NUM>, a camera <NUM>, a sensor <NUM>, a touch screen <NUM>, an audio outputter <NUM>, a driver <NUM>, a storage <NUM>, and a power supply <NUM>. In addition, the refrigerator <NUM> may further include a sensor (e.g., temperature sensor (not illustrated) or overheat sensor (not illustrated), and the like, <NUM>).

The controller <NUM> may include the processor <NUM>. The controller <NUM> may store a non-volatile memory which includes ROM <NUM> that stores a control program for controlling the refrigerator <NUM> and a volatile memory which includes RAM <NUM> that stores a signal or data inputted from outside the refrigerator <NUM> or is used as a storage area with respect to various operations performed on the refrigerator <NUM>. The RAM <NUM> may be used as a storage area with respect to control information received from an external source, operation information of the refrigerator <NUM> or state information of the refrigerator <NUM>.

The controller <NUM> serves to control a signal flow between the overall operation of the refrigerator <NUM> and the internal components <NUM> to <NUM> of the refrigerator <NUM> and process data. The controller <NUM> may control the power supply to the internal elements <NUM> to <NUM> by using the power supply <NUM>. The processor <NUM>, the ROM <NUM>, and the RAM <NUM> may be connected with one another by a bus.

The controller <NUM> may control the communication interface <NUM>, the microphone <NUM>, the camera <NUM>, the sensor <NUM>, the touch screen <NUM>, the audio outputter <NUM>, the driver <NUM>, the storage <NUM>, and the power supply <NUM>.

In an example embodiment, a refrigerator includes a main body having a storage, a first door having a display on a front surface thereof and hinged on the main body to be rotated on the front side of one side of the storage, a first camera configured to photograph food which is an object to be recognized, a second door which is hinged on the main body so that it is rotated on a front side of the other side of the storage, and a controller configured to control the display and the first camera. The controller controls, by using a food image photographed by the first camera a food name corresponding to the food image to be displayed on the display.

The controller <NUM> may control the display to display a screen of a food management application in response to the user approach.

The controller <NUM> may control an image of the food captured through the first camera to be recognized through a learning model by using the food management application.

The refrigerator may further include a storage configured to store a training data corresponding to the learning model. The controller <NUM> may control an image of the food to be recognized through the learning model.

The controller <NUM> may control to attempt recognition of the photographed image of the learning model in the determined order of the storage configured to store the training data, an external server, a server of an external store, and a web search engine.

The controller <NUM> may control to perform a preview corresponding to the food by using the first camera.

The controller <NUM> may control food information corresponding to the food image to be displayed on the display.

The controller <NUM> may control a storage position within the storage of the food in the food information to be displayed through the display by using a second camera to photograph the storage.

In a case in which the food is stored in the storage position within the storage, the controller <NUM> may control a storage period of the stored food to be displayed on the display.

The controller <NUM> may control an arrival of a storage period limit of the food to be displayed on the display by using a storage period of the food information.

The controller <NUM> may control an arrival of the storage period limit of the food to be transmitted to an external apparatus via the communication interface.

In another example embodiment, a refrigerator includes a display which is hinged to the main body to rotate on a front side of a storage, the display being disposed on a front surface of a door, a first camera which photographs a food which is an object to be recognized, the first camera being positioned on a front side of the other side of the storage, a second door which is hinged to the main body so that it is rotated on a front side of the other side of the storage, and a controller which controls the display and the first camera and controls a food name corresponding to the food image to be displayed on the display by using a food image photographed by the first camera.

In another example embodiment, a refrigerator includes a main body having a storage, a storage configured to store a first comparison food, a door which is hinged to the main body to rotate on a front side of one side of the storage, the door having a display on a front surface thereof, a camera configured to photograph food which is an object to be recognized, the camera being positioned on a front surface of the door, and a controller configured to control the storage, the display and the camera. The controller preferentially recognizes a food image photographed by the camera by using the first comparison food image of the storage according to a food recognition search result, and controls a food name corresponding to the photographed food image on the display.

A refrigerator may further include a communication interface. The controller <NUM> may recognize a food image photographed by the camera by using a second comparison food image stored in a storage of an external server connected via the communication interface according to the food recognition search order, and controls a food name corresponding to the photographed food image to be displayed on the display.

The controller <NUM> may recognize a food image photographed by the camera by using a third comparison food image stored in a storage of an store server connected via the communication interface according to the food recognition search order, and controls a food name corresponding to the photographed food image to be displayed on the display.

The controller <NUM> may recognize a food image photographed by the camera by using a search engine through the communication interface according to the food recognition search result, and controls a food name corresponding to the photographed food image to be displayed on the display.

According to an example embodiment, the term "controller of a refrigerator" as used herein may be a concept which includes the processor <NUM>, the ROM <NUM>, and the RAM <NUM>. In addition, "a controller of a refrigerator" may refer to the processor <NUM>.

The communication interface <NUM> may connect with an external devices <NUM> to <NUM> via a mobile communication network, a wireless LAN communication network, or a near field communication network by using one antenna or two or more antennas under the control of the controller <NUM>. The wireless LAN communication may be wirelessly connected with an access point (AP) at a location where the AP is installed under the control of the controller <NUM>. For example, the wireless LAN communication may include a Wi-Fi communication. The near field communication may include a Bluetooth communication, a Bluetooth low energy communication, an infrared data association (IrDA) communication, a ultra-wideband (UWB) communication, a magnetic security transmission (MST) communication and/or an NFC communication.

According to the various example embodiments, the term "communication interface" may be connect with an external device via a mobile communication, a wireless LAN communication and/or near field communication.

A microphone <NUM> may generate (or convert) a voice or sound which is received from an external source to an electric signal under the control of the controller <NUM>. The electric signal which is generated by the microphone <NUM> may be converted in an audio codec under the controller of the controller <NUM> and stored in the storage <NUM> or outputted through the audio outputter <NUM>.

The camera <NUM> may photograph a still image or record a video under the control of the controller <NUM>. The camera <NUM> may photograph a still image or record a video for registration and/or management of food. The camera <NUM> may photograph an iris of the user under the control of the controller <NUM>.

The camera <NUM> includes a front camera (or a first camera, <NUM>) which is positioned on a front side of the refrigerator <NUM> and an inner camera (or a second camera, <NUM>) which is positioned inside the refrigerator <NUM>. In addition, an iris photographing camera (not illustrated) which photographs an iris of a user may be positioned on one side of the front camera <NUM>.

The front camera (or an optical axis of the front camera <NUM>) may be inclined at a set angle (e.g., less than <NUM>°) towards a front side (e.g., +y-axis direction) based on the front side (or surface) of the door <NUM>. The above-mentioned setting angle may be <NUM> degrees or less or may be <NUM> degrees or more. In addition, the above-mentioned setting angle may be <NUM> degrees or less or may be <NUM> degrees or more. In a case in which the front camera <NUM> faces the front side (e.g., +y-axis direction), the user needs to lift food which is an object to be photographed into a photographing range (e.g., a view angle) of the front camera <NUM>. The front camera <NUM> which is inclined at a set angle can easily (or with a little power) photograph food which is an object to be photographed under the control of the controller <NUM>.

One of the front camera <NUM> and the inner camera <NUM> may include an auxiliary (e.g., flash (not illustrated) which provides an amount of light required for photographing. In addition, an auxiliary light source (e.g., an LED for iris recognition, not illustrated) required for photographing an iris may be positioned on a front surface of the refrigerator <NUM>.

One front camera <NUM> or a plurality of front cameras <NUM> may be included. One or more inner cameras <NUM> may be positioned on a rear surface of the door <NUM> to <NUM> which faces the storage <NUM> to <NUM> according to an area of the storage. For example, one or more inner cameras <NUM> may be positioned on a rear surface of the door <NUM> and <NUM> which faces the storage <NUM>. One or more inner cameras <NUM> may be positioned on a rear surface of the door <NUM> which faces the storage <NUM>. In addition, one or more inner cameras <NUM> may be positioned on a rear surface of the door <NUM> which faces the storage <NUM>.

The controller <NUM> may control a photographed image to be stored in the storage <NUM> through one of the front camera <NUM> and the inner camera <NUM>.

The sensor <NUM> may detect a peripheral state (e.g., illuminance) of the refrigerator <NUM> and/or an internal state (e.g., temperature of storage chamber) of the refrigerator <NUM> through one or a plurality of sensors.

The sensor <NUM> may include a proximity sensor for detecting whether a user approaches to the refrigerator <NUM> and one or more temperature sensors <NUM> for detecting a temperature of a storage (e.g., a freezer, a cold room, a vegetable room, and the like) of the refrigerator <NUM>. In addition, the sensor <NUM> may include an illuminance sensor (not illustrated) which detects an amount of light in the periphery of the refrigerator <NUM> corresponding to a change of brightness. One of the proximity sensor <NUM> and the illuminance sensor (not illustrated) from among the sensor <NUM> may be positioned on a front surface of the door of the refrigerator <NUM>. In addition, the temperature sensor <NUM> may be positioned indoors to correspond to the storage.

It would be easily understood by those skilled in the art that a sensor type included in the sensor <NUM> may be added, modified or deleted according to a performance of the refrigerator <NUM>.

A touch screen (or display) may provide a graphical user interface (GUI) corresponding to various services (e.g., data transmission, video call, data transmission, broadcast reception, viewing of a video content, electronic payment including mobile payment, or the like). In an example embodiment, a touch screen may be a concept which includes a display.

The touch screen <NUM> may include a touch panel <NUM> which receives a user input (e.g., touch) and a display panel <NUM> for displaying a screen. In the touch screen <NUM>, the display panel <NUM> and the touch panel <NUM> may be implemented as integral-type (e.g., in-cell type touch screen or on-cell type touch screen).

The touch screen <NUM> may include an edge touch panel (not illustrated) which receives a user input and an edge display panel (not illustrated) for displaying a screen. The edge touch panel and the edge display panel may be implemented as integral-type as described above.

The touch screen <NUM> may transmit an analog signal corresponding to a single touch or multi touch which is input through a home screen (not illustrated) or a GUI to a touch screen controller (not illustrated). The touch screen <NUM> may receive a single touch or multi-touch through a body (e.g., fingers including a thumb) of the user or an input pen (e.g., stylus, not illustrated).

According to an example embodiment, a touch screen (or a display, <NUM>) may output a visual feedback corresponding to an indication of a storage position within a storage of the identified food. In an example embodiment, the display may be a concept which includes the touch screen <NUM>.

A touch screen controller (not illustrated) converts an analog signal corresponding to a touch (single touch or multi-touch) received from the touch screen <NUM> into a digital signal, and transmits the digital signal to the controller <NUM>. The controller <NUM> may calculate a position (e.g., x coordinate and y coordinate) of a touch received by the touch screen <NUM> by using the digital signal received from the touch screen controller (not illustrated).

The controller <NUM> may control the touch screen <NUM> by using the digital signal received from the touch screen controller (not illustrated). For example, in response to a touch being input, the controller <NUM> may display a shortcut icon (or referred to as icon) selected from among shortcut icons corresponding to applications displayed on the touch screen <NUM> distinctively from other shortcut icons that are not selected, or may execute an application (e.g., a video application) corresponding to the selected shortcut icon and display a video application screen on the touch screen <NUM>.

The audio outputter <NUM> may output sound corresponding to various signals (e.g., a wireless signal, a broadcast signal, an audio source, a video file or photographing of a photo) decoded by an audio codec under the control of the controller <NUM>. The audio outputter <NUM> may include one or more speakers.

One or more audio outputters <NUM> may be positioned on one side and/or lateral side of the refrigerator <NUM>. One or more audio outputters <NUM> may be positioned to be adjacent to a touch screen (or a display, <NUM>) positioned on a front surface of the refrigerator. In addition, one or more audio outputters <NUM> may be positioned to be adjacent to the front camera <NUM> positioned on a front surface of the refrigerator.

According to an example embodiment, the audio outputter <NUM> may output an auditory feedback corresponding to an indication of a storage position within a storage of the identified food.

The driver <NUM> may include a compressor <NUM>, a fan <NUM>, a filter <NUM> or a heater <NUM> which is operated under the control of the controller <NUM>. The driver <NUM> may further include a lighting (not illustrated) or a smell removing apparatus (not illustrated).

The compressor <NUM> compresses a refrigerant which is an operating fluid of a freezing cycle under the control of the controller <NUM>. The freezing cycle includes a condenser (not illustrated) which transforms a gas refrigerant compressed by the compressor <NUM> into a liquid refrigerant, an expander (not illustrated) which reduces a pressure in the liquid refrigerant, and an evaporator (not illustrated) which gasifies the pressure-reduced liquid refrigerant. The controller <NUM> may control a temperature in the storage through gasification of a liquid refrigerant. In addition, the refrigerator <NUM> may control a temperature of a storage through a peltier element (not illustrated) using a peltier effect, a magnetic cooling apparatus (not illustrated) using a magnetocaloric effect.

The fan <NUM> may circulate an outside air under the control of the controller <NUM>. An air which is heated by a cooling cycle may be cooled from heat exchange through an outside air.

The filter <NUM> may include an air purifier. The filter <NUM> may be floated within the storage under the control of the controller <NUM> or may sterilize (or remove) the attached virus. The filter <NUM> may include an icon sterilization purifier.

The heater <NUM> removes frost which occurs under the control of the controller <NUM>. The heater <NUM> may include a defrost heater.

The storage <NUM> may store a signal or data (e.g., corresponding to food management (or food recognition) that is inputted or outputted to correspond to an operation of elements <NUM> to <NUM> under the control of the controller <NUM>. The storage <NUM> may store a control program for controlling the refrigerator <NUM> or the controller <NUM> and a graphical user interface (GUI) relating to an application (e.g., a food management application, a food recognition application, or the like) provided from the manufacturer and downloaded from an outside source, an image corresponding to the GUI, user information, a document, databases or relevant data.

The storage <NUM> may store an image (or video) of food photographed in the camera <NUM>. A food image <NUM> may be a concept which includes a binary file including one or more images displayed on the touch screen <NUM> and one or more images stored in the storage <NUM>. In addition, the storage <NUM> may store various types of comparison food images corresponding to a comparison of the photographed food images (or videos). The various types of comparison food images may include many (e.g., tens of or hundreds of) food images for each type of food. In a case in which the photographed food image (or video) is a carrot, the comparison food image may be the comparison food image may include images of a carrot, such as various shapes, sizes, colors, and wash states of a carrot and whether the carrot is packed or not.

The storage <NUM> may store various training data corresponding to food recognition to recognize (or distinguish) a photographed food image. For example, an image of a carrot including various shapes, sizes, colors or wash states of the carrot and whether the carrot is packed or not may be stored in the storage <NUM> as training data to recognize a carrot. In addition to fruit, the training data corresponding to food that is edible, drinkable or cookable by a user, such as vegetables, meat, seafood and the like may be stored in the storage <NUM>. In addition, the storage <NUM> may store a learning model to recognize a photographed food image.

The storage <NUM> may store an image (or video) of food photographed by the camera <NUM> and store a food recognition application (or an additional application using a food recognition algorithm) which distinguishes food to be identified from the photographed food image through machine learning. All of a food recognition application may be stored in the storage. Some of a food recognition application may be stored in an external apparatus (e.g., a server or a portable device). Some of a food recognition application may be stored in the storage <NUM>. In addition, some of the training data corresponding to food recognition may be stored in the storage <NUM>. Alternatively, some of the training data corresponding to food recognition may be stored in an external apparatus (e.g., a server or a portable device).

The controller <NUM> may recognize (or distinguish) food from an image of food photographed in the camera <NUM> by using a food recognition application.

The storage <NUM> may store a visual feedback (e.g., a video source and the like) which is outputted to correspond to an indication of a storage position within a storage of the identified food and is recognizable by a user and an auditory feedback (e.g., sound source and the like) which is outputted from the audio outputter <NUM> and is recognizable by a user.

The storage <NUM> may store a feedback providing time (e.g., <NUM>) of feedback provided to a user.

In an example embodiment, the term "storage" includes a storage <NUM>, ROM <NUM> within the controller <NUM>, and a memory card (e.g., a micro SD card and the like, not illustrated) which is mountable to RAM <NUM> or a slot (not illustrated) of the refrigerator <NUM>. The storage <NUM> may include a non-volatile memory, a volatile memory, a hard disk drive (HDD) or a solid state drive (SSD).

The power supply <NUM> may supply power to elements <NUM> to <NUM> of the refrigerator <NUM> under the control of the controller <NUM>. The power supply <NUM> may supply power inputted from an external power source to each element of the refrigerator <NUM> via a power code (not illustrated) under the control of the controller <NUM>.

It would be easily understood by those skilled in the art that at least one element of the refrigerator <NUM> illustrated in FIGS. 1A, <FIG> and <FIG> can be added, deleted or modified in response to performance of the refrigerator <NUM>.

<FIG> illustrates a flowchart of a food management method of a refrigerator having a front camera and a display, according to an embodiment of the present disclosure.

<FIG> illustrate diagrams of an example food management method of a refrigerator having a front camera and a display, according to embodiments of the present disclosure.

At step S310 of <FIG>, a user approach is detected.

Referring to <FIG>, a user holding a food <NUM> is positioned on a front side of the refrigerator <NUM>. The food <NUM> may include a packed state or an unpacked state. With regard to the food in a packed state, the packed food may include a container 1a accommodating the food <NUM> and a packing material 1b packing food accommodated in the container. Additional information of food (e.g., a trademark (not illustrated), a price (not illustrated) or a barcode (not illustrated)) may be positioned on a surface of the packing material 1b. The above-described additional information of food may be recognized as well in the process of recognizing the food through the front camera <NUM>.

One content (e.g., a weather widget or a time widget) from among various contents may be displayed on the touch screen <NUM> positioned on a front surface of the door of the refrigerator <NUM>.

The refrigerator <NUM> may include a user approach detector for detecting a user positioned on a front side. The user approach detector may include one of a microphone <NUM> positioned on a front surface of the door, a front camera <NUM> inclined at a set angle, and a proximity sensor <NUM>. In addition, the user approach detector may include a combination of the microphone <NUM> at a front surface of the door, the front camera <NUM> inclined at a set angle, and a proximity sensor <NUM>.

The controller <NUM> of the refrigerator <NUM> may detect an approaching of the user to the refrigerator by using the user approach detector. The microphone <NUM> may receive a user voice that is uttered. The controller <NUM> may distinguish (or authenticate) the user by recognizing the user voice.

The front camera <NUM> inclined at a set angle may photograph an appearance (or a user motion or a user presence) of the user on a front side of the refrigerator. The controller <NUM> may distinguish (or authenticate) the user by recognizing a photographed image (e.g., face). The proximity sensor <NUM> may detect an approaching of the user from the front side of the refrigerator <NUM>.

The user approach detector may detect a user approach through one sensor or may detect a user approach through a plurality of sensors. In addition, the user approach detector may distinguish (e.g., through a microphone or a camera) the user approaching the refrigerator through one sensor, or may distinguish (e.g., through a combination of the proximity sensor and a camera, a combination of the proximity sensor and a microphone, and the like) the user approaching through a combination of a plurality of sensors.

The user approach detector may output an electrical signal corresponding to detection of a user approach to the controller <NUM>. In addition, the user approach detector may output an electrical signal corresponding to detection of a user approach to the storage <NUM>.

At step S320 of <FIG>, a screen of a food management application is displayed on a display.

Referring to <FIG>, the controller <NUM> may display a screen of a food management application on the touch screen <NUM> in response to detection of a user approach. The controller <NUM> may change (e.g., from a widget screen 170a to a food management application screen <NUM>) a displayed content in response to detection of a user approach.

The controller <NUM> may display a screen of a food management application in which food may be registered in response to detection of a user approach. The screen of the food management application screen <NUM> may include a title area <NUM> and a display area <NUM> of food stored in a storage (e.g., a cold room). In <FIG>, the "Register food 301a" is displayed on the title area <NUM>. An image displayed on the food management area <NUM> may be photographed by an inner camera <NUM>.

The user may touch the Register food 301a of the title area <NUM> to photograph food (e.g., a carrot, <NUM>) held by the user, which is an object to be photographed, through the front camera <NUM> inclined at a set angle.

At step S330 of <FIG>, food is photographed in the front camera.

Referring to <FIG>, the controller <NUM> may move the front camera <NUM> inclined at a set angle, perform a preview (e.g., a video and a preview, 302a) corresponding to the food <NUM> to be photographed, and display the same on the touch screen170 in response to the user touching the food and the like 301a. The preview 302a may be displayed on a partial area (e.g., more than <NUM>% and less than <NUM>% of the display area) of the touch screen170 or may be displayed on the entire area of the touch screen170. The user may identify a position (e.g., three-dimensional position) appropriate for the front camera <NUM> to photograph food through the preview 302a.

In a case in which the preview 302a is displayed on the touch screen170 and a user input is not inputted for a preset time (e.g., one second, modifiable), the controller <NUM> may continuously photograph the food <NUM> by using the front camera <NUM>.

Referring to <FIG>, the controller <NUM> may operate the front camera <NUM> in response to a user touch of the Register food 301a. The front camera <NUM> may continuously photograph (e.g., several (or tens of) photos per second or several (tens of) photos at intervals of <NUM> second, the time interval may be modified) a food <NUM> to be photographed under the control of the controller <NUM>.

The touch screen170 may display a title area ('display photographed image 301a', <NUM>) and 'photographed image displayed area <NUM>' under the control of the controller <NUM>. The touch screen170 may display a plurality of photographed food images <NUM>, 304a to 304j on the photographed image display area <NUM> under the control of the controller <NUM>. The number of the displayed food images <NUM> is only an example, and may be modified according to a type and/or size of food.

The photographed food image <NUM> may be stored in the storage <NUM> under the control of the controller <NUM>.

At step S340 of <FIG>, a photographed food is identify.

Referring to <FIG> and <FIG>, the controller <NUM> may recognize food (or food name, <NUM>) through the photographed food image <NUM>. The controller <NUM> may recognize the photographed food image <NUM> and identify food (or food name, <NUM>). The controller <NUM> may recognize the photographed food image <NUM> through machine learning and identify food (or food name, <NUM>). The controller <NUM> recognize the photographed food image <NUM> through deep learning using a neural network model and identify food (or food name, <NUM>).

The controller <NUM> may recognize the food image <NUM> through a food recognition application (or a food management application, not illustrated) being executed and identify a name (e.g., a carrot, an apple and the like) of the food <NUM>. The controller <NUM> may identify a name of the food <NUM> by recognizing an image of food through a food recognition application (or a food management application) executed in response to a photographing of the food <NUM> of the front camera <NUM> inclined at a set angle.

The controller <NUM> may display "processing" on the touch screen170 in response to a food recognition process of the food recognition application.

The food recognition application may identify food (or a name of food, <NUM>) by using a learning model generated through learning of a training data and the food image <NUM>. The training data may include a supervised learning data and an unsupervised learning data. The machine learning using the supervised learning data may execute machine learning by using a food name (e.g., "carrot") corresponding to the supervised learning data and a food image (e.g., images of carrots of various sizes and numbers). The machine learning using the unsupervised learning data may execute machine learning by using a food image (e.g., images of carrots of various sizes and numbers).

The food recognition application may find a rule or a pattern from a stored training data. The food recognition application may find a rule or a pattern from the training data through a learner. A learning model may be generated through a rule or pattern found by the learner. The food recognition application may identify food through a generated learning model and the food image <NUM>.

The learning data described above may be stored in an external server <NUM> which is connected to the storage <NUM> and the refrigerator <NUM> via a communication interface. The learning data described above may be regularly updated to the storage <NUM> and stored in (or updated to) an external server <NUM> connected to the refrigerator <NUM> via a communication interface. A storage (not illustrated) of a server <NUM> storing a training data may include a storage of a server (not illustrated) of a manufacturer of the refrigerator <NUM>, a storage of a server (not illustrated) which is capable of downloading a food recognition application, and a storage of an application server (not illustrated) which is capable of downloading an application executable in the refrigerator.

For machine learning, the food recognition application may preferentially primarily use (or learn) a training data stored in the storage <NUM> of the refrigerator <NUM> and then, secondly use (or learn) a training data stored in the storage (not illustrated) of the server <NUM>. The training data which is used for food recognition of a learning model may be first used in the storage <NUM> of the refrigerator <NUM> and then, may be secondly used in the storage (not illustrated) of the server <NUM>. The training data of the storage <NUM> described above may be periodically updated. In addition, the training data stored in the storage of the server <NUM> may be periodically updated.

In a case in which the food <NUM> is not identified through first image recognition using the storage <NUM> in the food recognition application, the controller <NUM> may proceed with a second image recognition by using the training data stored in the storage of the server <NUM> via the communication interface <NUM>. In addition, in a case in which the food <NUM> is not identified through first image recognition using the storage <NUM> in the food recognition application, the controller <NUM> may execute a second image recognition by using search history or frequently-searched images of the user (or the refrigerator <NUM>) stored in the storage of the server <NUM> via the communication interface <NUM>.

In a case in which the food <NUM> is not identified through a second image recognition of the communication interface <NUM> and the storage of the server <NUM> in the food recognition application, the controller <NUM> may search the storage (not illustrated) of the server (not illustrated) of a store (or a mart) that sold the food via the communication interface <NUM> and execute a third image recognition.

In a case in which the food <NUM> is not identified through a third image recognition using the communication interface <NUM> and the storage (not illustrated) of the server (not illustrated) of a mart (or a store) in the food recognition application, the controller <NUM> may execute a fourth image recognition through performing search through the communication interface <NUM> and a search engine.

The food recognition search order described above (e.g., first, a storage <NUM>, a storage of a server <NUM>, a storage (not illustrated) of a server (not illustrated) of a mart (or a store) and last, a search engine, which may be called a "food recognition search order") may be changed (e.g., an attempt may be made in a search engine after an attempt is made in the storage <NUM>) by the controller <NUM>.

The controller <NUM> may identify the food <NUM> according to a food recognition application and a food recognition search order. In a case in which a food recognition is identified, the controller <NUM> may not display a displayed "processing" <NUM> on the touch screen170.

The storage <NUM> may store the identified food (or a food name) under the control of the controller <NUM>.

Referring to <FIG>, a food recognition application according to another example embodiment may analyze (or identify) food (or a food name, <NUM>) by using a feature (or a feature point and image feature) of the food image <NUM>, at operation S610 of <FIG>. The feature of an image may include a feature point of the image and/or an amount of feature of the image. The feature amount of the image may be an area in which a luminance value between consecutive pixels in the image is greater than a set value (e.g., <NUM>, modifiable). The feature amount of the image is a value indicating a feature value of the image in a vector value.

The food recognition application may compare a feature point of the image by using the feature amount of the image. The controller <NUM> may compare (e.g., similarity between feature points of a plurality of images) the feature points of the images by using the feature amount of the image calculated through the food recognition application. Various methods of extracting a feature point of an image and a method of calculating a feature amount of an image are described above and thus, the detailed description thereof will be omitted.

The food recognition application may compare the feature amount of the image corresponding to the food image <NUM> with the feature amount of the image corresponding to a comparison food image pre-stored in the storage <NUM>.

The pre-stored comparison food image may be stored in the storage <NUM> and an external server <NUM> connected to the refrigerator <NUM> via a communication interface. A storage (not illustrated) of a server <NUM> storing a pre-stored comparison food image may include a storage of a server (not illustrated) of a manufacturer of the refrigerator <NUM>, a storage of a server (not illustrated) which is capable of downloading a food recognition application, and a storage of an application server (not illustrated) which is capable of downloading an application executable in the refrigerator.

A pre-stored comparison food image for food recognition of a food recognition application may perform a first comparison (operation S620 of <FIG>) prior to comparing the storage <NUM> of the refrigerator <NUM> and then, perform a second comparison (operation S630 of <FIG>) of the storage of the server <NUM>. The pre-stored comparison food image of the storage <NUM> described above may be periodically (e.g., time, day, week, month and the like, modifiable) updated. A comparison food image pre-stored in the storage of the server <NUM> may be periodically (e.g., time, day, week, month and the like, modifiable) updated as well.

In a case in which the food <NUM> is not identified through first image recognition using a comparison food image pre-stored in the storage <NUM> in the food recognition application, the controller <NUM> may proceed with a second image recognition by using the comparison food image pre-stored in the storage of the server stored in the storage of the server <NUM> via the communication interface <NUM>. In addition, in a case in which the food <NUM> is not identified through first image recognition using a comparison food image pre-stored in the storage <NUM> in the food recognition application, the controller <NUM> may execute a second image recognition by using search history or frequently-searched images of the user (or the refrigerator <NUM>) stored in the storage of the server <NUM> via the communication interface <NUM>.

In a case in which the food <NUM> is not identified through a second image recognition of the communication interface <NUM> and the storage of the server <NUM> in the food recognition application, the controller <NUM> may search (operation S640 of <FIG>) for a comparison food image pre-stored in the storage (not illustrated) of the server (not illustrated) of a mart (or a store) that sold the food via the communication interface <NUM> and execute a third image recognition.

In a case in which the food <NUM> is not identified through a third image recognition using the communication interface <NUM> and the storage (not illustrated) of the server (not illustrated) of a mart (or a store) in the food recognition application, the controller <NUM> may execute a fourth image recognition through performing search (operation S650 of <FIG>) through the communication interface <NUM> and a search engine.

The food recognition search order described above (e.g., first, a storage <NUM>, a storage of a server <NUM>, a storage (not illustrated) of a server (not illustrated) of a mart (or a store) and last, a search engine (or a web search engine), which may be called a "food recognition search order") may be changed (e.g., an attempt may be made in a search engine after an attempt is made in the storage <NUM>) by the controller <NUM>. A storage of the server <NUM> described above and a storage (not illustrated) of a server (not illustrated) of a store (or a mart, a department store, an online shopping mall and the like) may be called an external storage.

The controller <NUM> may identify the food <NUM> according to a food recognition application and a food recognition search order, at operation S660 of <FIG>. In a case in which food is identified, the controller <NUM> may not display a displayed "processing" <NUM> on the touch screen170.

In a case in which the food <NUM> is not identified, the touch screen170 may display "food not identified (or 'food name identified')" under the control of the controller <NUM>. In a case in which the food <NUM> is not identified, the touch screen <NUM> may display a pop-up (not illustrated) corresponding to the "food not identified" under the control of the controller <NUM>. In addition, a voice and/or sound corresponding to the "food not identified" may be outputted through the audio outputter <NUM>.

At operations S620 to S650 of <FIG>, in a case in which the food <NUM> is not identified, a food management method of a refrigerator is ended.

At operation S660 of <FIG>, in a case in which the food <NUM> is identified, the operation S350 of <FIG> is performed.

At operation S350 of <FIG>, food information corresponding to the identified food is displayed.

Referring to <FIG>, the controller <NUM> may display the food information 301d and display area <NUM> corresponding to the identified food on a food information display area <NUM>. Items included in the food information <NUM> may, for example, include a food name (e.g., "carrot", 306a), a recommended location at which food is stored (e.g., "moisture care vegetable room in the right cold room", 306b), a storage date at which the identified food is stored (e.g., "Jan. <NUM>, <NUM>, <NUM>:<NUM>:<NUM> PM", 306c) and a storage period limit of food (e.g., "until Jan. <NUM>, <NUM>, <NUM>:<NUM> PM", 306d).

The controller <NUM> may add food additional information which is positioned in the packing material 1b of the packed food to the food information. The controller <NUM> may add additional information of food which is positioned at the packing material 1b of the packed food to the food information. The food information described above is only an example, and may further include a place of origin, whether discount is applied and/or a user input (e.g., "let's eat" and the like).

In a case in which the number of items included in the food information <NUM> is greater than the number of items that may be displayed on the touch screen <NUM>, the user may scroll up or down the food information <NUM> through a user input (e.g., touch, voice, motion and the like).

The storage <NUM> may store the displayed (displayable) food information under the control of the controller <NUM>.

At operation S360 of <FIG>, food stored in the storage is displayed in correspondence with the food information.

Referring to <FIG>, the controller <NUM> may provide a storage location within the storage of the food according to the food information. The touch screen <NUM> may display a title area ("food position 301e)", <NUM>) and a food storage position <NUM> under the control of the controller <NUM>.

The food storage position <NUM> within the storage of the food <NUM> may be provided to the user through at least one of an auditory feedback which is outputted from the audio outputter <NUM> and a visual feedback displayed from the touch screen <NUM>. The controller <NUM> may output the storage location (11d2, see <FIG>) within the storage of the food <NUM> as a sound (e.g., "moisture care vegetable room in right cold room (11d2)") according to the food information.

The controller <NUM> may photograph the storage location 11d2 within the storage of the food <NUM> through the inner camera <NUM> according to the food information and distinguishably display the photographed storage location on the touch screen170. In addition, the controller <NUM> may reduce (or briefly illustrate <FIG>) the storage location 11d2 within the storage of the food <NUM> according to the food information and distinguishably (adjacent (between the location 306b and the storage date 306c) to the location 306b within the food information <NUM>) display the reduced storage location on the touch screen170.

Referring to <FIG>, the food <NUM> stored in the storage is displayed.

The user may control to open the door <NUM> according to the displayed food information and to accommodate the food <NUM> in the moisture care vegetable room 11d2 in right cold room 11d2. The controller <NUM> may detect opening/closing of the door <NUM> by using a sensor (not illustrated).

When the door <NUM> is closing, the touch screen170 may display the food <NUM> accommodated in the moisture care vegetable room of right cold room 11d2 by a photographing of the inner camera <NUM> under the control of the controller <NUM>. The food <NUM> stored in the moisture care vegetable room in right cold room 11d2 and the storage period <NUM> of another food stored in the storage are respectively displayed. In addition, the food <NUM> stored in the moisture care vegetable room in right cold room 11d2 and the storage 308a may be displayed together.

The controller <NUM> may provide a user with feedback corresponding to a storing (or storing and displaying) of the food <NUM> within the storage according to the food information <NUM>.

The provided additional feedback may be provided in at least one of a visual feedback and an auditory feedback. In addition, the controller <NUM> may provide the user with a combination of the visual feedback and the auditory feedback.

The visual feedback may include a visual effect (e.g., an additional image or an animation effect like fade in applied to the additional image, not illustrated) corresponding to a storing (or storing and displaying) of the food <NUM> within the storage according to the food information <NUM>. The controller <NUM> may display the visual feedback on the touch screen170.

The auditory feedback may include sound that corresponds to a storing (or storing and displaying) of the food <NUM> within the storage according to the food information <NUM>. The controller <NUM> may output auditory feedback via the audio outputter <NUM>. A feedback providing time (e.g., <NUM>) provided to the user may be changed through settings (not illustrated).

At operation S350 of <FIG>, a storage period limit of the stored food is reached.

The controller <NUM> may periodically identify the storage period of the food stored in the storage by using the food information. The controller <NUM> may identify the arrival of the storage period limit of the food <NUM>. The arrival of the storage period limit may be one day (or two days, may be modified according to setting) before the last day of the storage period. The controller <NUM> may compare the present day (Jan. <NUM>, '<NUM>) with the stored food information <NUM> and identify the arrival of the storage period limit. The controller <NUM> may identify the arrival of the storage period limit of the food <NUM> based on the present day and the storage period limit 306d of the food information <NUM>.

At operation S370 of <FIG>, in a case in which the storage period limit of the food <NUM> has arrived, the operations S380 and S390 of <FIG> are performed. In a case in which the storage period limit of the food <NUM> has not arrived, the process proceeds to a process before the step S370 of <FIG>.

At operation S380 of <FIG>, an arrival of the storage period limit is displayed on the display.

Referring to <FIG>, an arrival <NUM> of storage period limit of the food <NUM> is displayed on the touch screen170. The arrival <NUM> of storage period limit of the food <NUM> may be displayed as distinguishable from the storage period <NUM> of another food.

In a case in which the user touches one from among the arrival <NUM> of storage period limit and the food <NUM>, detailed information 309a (see <FIG>) corresponding to the arrival of storage period limit of the food <NUM> may be displayed on the touch screen170. A pop up (overlay or overlap) of the detailed information 309a may be displayed.

At operation S380 of <FIG>, in a case in which an arrival of storage period limit is displayed on the display, a food management method of the refrigerator is ended.

At operation S390 of <FIG>, an arrival of storage period limit is notified to an external apparatus.

Referring to <FIG>, a refrigerator management application 227a is displayed on the display <NUM> of the portable apparatus <NUM>. The portable apparatus <NUM> managing the refrigerator <NUM> may functionally connected with the refrigerator <NUM> via a communication interface (not illustrated). The refrigerator management application 227a executed in the external apparatus <NUM> managing the refrigerator <NUM> may functionally connected with the refrigerator <NUM> via a communication interface (not illustrated).

The external apparatus <NUM> may receive an arrival of storage period limit of the food <NUM> from the refrigerator <NUM> via the communication interface. The portable apparatus <NUM> may receive a notification message (or detailed information) corresponding to the arrival of storage period limit of the food <NUM> from the refrigerator <NUM> via the communication interface.

The portable apparatus <NUM> may display the received notification message on the display <NUM>. The external apparatus <NUM> may display a pop-up of (or overlay or overlap) the received notification message over the refrigerator management application displayed on the display <NUM>. The notification message 309a may include a recommended recipe 309a1 of the food <NUM> of which the storage period limit has arrived.

At operation S390 of <FIG>, in a case in which the arrival of storage period limit is notified to an external apparatus. The food management method of the refrigerator is ended.

The methods according to exemplary embodiments of the present disclosure may be implemented as a program instruction type that may be performed through various computer units and may be recorded in a non-transitory computer readable medium. The computer-readable medium may include a program command, a data file, a data structure or the like, alone or a combination thereof. For example, the computer-readable medium may be stored in a volatile or non-volatile storage device such as a ROM, a memory such as a RAM, a memory chip, and a device or an integrated circuit, or a storage medium which may be read with a machine (for example, computer) simultaneously with being optically or magnetically recorded like a CD, a DVD, a magnetic disk, a magnetic tape, or the like, regardless of whether it is deleted or again recorded.

In additional, the computer-readable medium may be stored in a computer-readable storage medium of a server, and the computer program may be downloaded to a computing device via network.

According to an embodiment, the methods according to the various embodiments disclosed in the present document may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a purchaser. The computer program product may be distributed in a form of a storage medium (for example, a compact disc read only memory (CD-ROM)) that may be read by a device or online through an application store (for example, PLAYSTORE). In the case of the online distribution, at least portions of the computer program product may be at least temporarily stored in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server or be temporarily generated.

Claim 1:
A refrigerator (<NUM>), comprising:
a main body (<NUM>) including a storage (<NUM>, <NUM>, <NUM>);
a door (<NUM>) configured to be hinged to the main body (<NUM>) to rotate on a front side of one side of the storage (<NUM>, <NUM>, <NUM>) and include a display (<NUM>) on a front surface of the door (<NUM>);
characterized in that, the refrigerator (<NUM>) further comprises:
a first camera (<NUM>) positioned on the front surface of the door (<NUM>) and configured to photograph food (<NUM>);
a second camera configured to photograph an inside of the storage; and
a controller (<NUM>) configured to:
control the display (<NUM>),
control the first camera (<NUM>) to photograph a first image of the food,
control the second camera (<NUM>) to photograph a second image of the inside of the storage,
recognize a name of the food (306a) based on the first image, and
provide food information (301d, <NUM>) including the name of the food and a recommended storage location at which the food is stored on the display,
wherein the controller (<NUM>) is configured to control the display to display the second image tagged with the food information (301d, <NUM>).