Patent Description:
A technology is known which compares the feature amount of an object extracted from an image obtained by photographing the object with each of the feature amounts for collation pre-registered in an information dictionary and recognizes the category of the object according to the similarity degree therebetween. Besides, a store system is proposed which registers the sales of a commodity such as a fresh fruit or vegetable recognized using a technology relating to an object recognition.

Moreover, the surface color of a commodity is almost same to the color of the background depending on a type or category of the commodity when such a commodity is photographed. In this case, it is difficult to detect the boundary between the commodity and the background, resulting in that it is sometimes impossible to extract a feature amount a level of which the commodity can be recognized. Thus, it is considered that there is still room for improvement of recognition rate.

By taking a checkout system as an example, the information processing apparatus and the information processing method according to the present embodiment are described below with reference to the accompanying drawings. A store system is a checkout system (POS system) equipped with a POS (Point of Sales) terminal for the registration and settlement of a commodity in one transaction. The present embodiment is an example of the application to a checkout system introduced into a store such as a supermarket.

<FIG> is an oblique view exemplifying a checkout system <NUM>. As shown in <FIG>, a checkout system <NUM> comprises a POS terminal <NUM> for the registration and settlement of commodities in one transaction and a commodity reading apparatus <NUM> for reading information relating to the commodities. The POS terminal <NUM> is hereinafter used as an information processing apparatus according to the present embodiment.

The POS terminal <NUM> is mounted on a drawer <NUM> on a checkout counter <NUM>. The drawer <NUM> is opened under the control of the POS terminal <NUM>. A keyboard <NUM> that is press-operated by an operator (shop clerk) is arranged on the upper surface of the POS terminal <NUM>. A first display device <NUM> for displaying information to the operator is arranged behind the keyboard <NUM>. The first display device <NUM> displays information on a display surface 12a thereof. A touch panel <NUM> is laminated on the display surface 12a. A rotatable second display device <NUM> is vertically arranged at a position behind the first display device <NUM>. The second display device <NUM> displays information on the display surface 14a thereof. The second display device <NUM> is rotated so that the display surface 14a is oriented toward a customer to show information to the customer.

A long desk-shaped counter table <NUM> is arranged in an L shape adjacent to the checkout counter <NUM> on which the POS terminal <NUM> is placed. A receiving surface <NUM> is formed on the counter table <NUM>. A shopping basket <NUM> for accommodating a commodity G is placed on the receiving surface <NUM>. The shopping basket <NUM> includes a first shopping basket 60a which is carried by a customer and a second shopping basket 60b which is placed opposite to the first shopping basket 60a across the commodity reading apparatus <NUM> as shown in <FIG>. The shopping basket <NUM>, which is not limited to be basket-shaped, may be a tray or the like. Moreover, the shopping basket <NUM> (the second shopping basket 60b), which is not limited to be of a basket shape, may be a box-shaped or a bag-shaped.

The commodity reading apparatus <NUM>, which is connected with the POS terminal <NUM> to transmit/receive data to or from the POS terminal <NUM>, is arranged on the receiving surface <NUM> of the counter table <NUM>. The commodity reading apparatus <NUM> has a thin rectangular-shaped housing <NUM>. A reading window <NUM> is arranged on the front surface of the housing <NUM>. A displaying and operating unit <NUM> is arranged at the upper part of the housing <NUM>. A first display device <NUM>, acting as a display unit, on which a touch panel is laminated is arranged on the displaying and operating unit <NUM>. A keyboard <NUM> is arranged at the right side of the first display device <NUM>. The card reading slot <NUM> of a card reader is arranged at the right side of the keyboard <NUM>. A second display device <NUM> for providing information to the customer, standing opposite to the operator, is arranged at a left backside of the displaying and operating unit <NUM>.

The commodity reading apparatus <NUM> is provided with a commodity reading unit <NUM> (refer to <FIG>) which is equipped with an image capturing section <NUM> behind the reading window <NUM> and a distance measurement section <NUM> (refer to <FIG>).

The commodities G to be settled in one translation are accommodated in the first shopping basket 60a carried by the customer. The commodities G in the first shopping basket 60a are moved one by one to the second shopping basket 60b by the operator who operates the commodity reading apparatus <NUM>. When being moved, the commodity G faces the reading window <NUM> of the commodity reading apparatus <NUM>. At this time, the image capturing section <NUM> (refer to <FIG>) arranged in the reading window <NUM> photographs the commodity G, on the one hand, and the distance measurement section <NUM> (refer to <FIG>) arranged in the reading window <NUM> measures the distance from the image capturing section <NUM> to an object (commodity) located in an image capturing range.

In the commodity reading apparatus <NUM>, a screen is displayed on the displaying and operating unit <NUM> to specify which one of the commodities registered in a PLU file F1 (refer to <FIG>) described later is corresponding to the commodity G contained in the image captured by the image capturing section <NUM>. Moreover, the commodity reading apparatus <NUM> notifies the commodity ID of the specified commodity to the POS terminal <NUM>. In the POS terminal <NUM>, the information relating to the sales registration of the commodity corresponding to the commodity ID notified by the commodity reading apparatus <NUM>, such as the commodity category, the commodity name and the unit price of the commodity, are recorded in a sales master file (not shown) for sales registration.

<FIG> is a block diagram illustrating the hardware structures of the POS terminal <NUM> and the commodity reading apparatus <NUM>. The POS terminal <NUM> comprises a microcomputer <NUM> functioning as an information processing unit for carrying out an information processing. The microcomputer <NUM> is formed by connecting, via a bus line, a ROM (Read Only Memory) <NUM> and a RAM (Random Access Memory) <NUM> with a CPU (Central Processing Unit) <NUM> that carries out various operations to control each unit.

The keyboard <NUM>, the first display device <NUM>, the touch screen <NUM>, the second display device <NUM> and the drawer <NUM> are all connected with the CPU <NUM> of the POS terminal <NUM> via various input/output circuits (not shown) to be controlled by the CPU <NUM>.

The keyboard <NUM> includes a numeric keypad (ten keys) 11a on the upper surface of which numeric characters or arithmetic operators are displayed or printed, a temporary closing key 11b, a closing key 11c and so on.

The CPU <NUM> of the POS terminal <NUM> is connected with an HDD (Hard Disk Drive) <NUM> in which various programs and files are stored. When the POS terminal <NUM> is started, the programs and files stored in the HDD <NUM> are all or partially copied into the RAM <NUM> to be executed by the CPU <NUM>. An example of the program stored in the HDD <NUM> is an application program for a sales registration processing or information dictionary registration processing described later. An example of the file stored in the HDD <NUM> is a PLU file F1.

The PLU file F1 is a commodity file in which information relating to the sales registration of each commodity G displayed in a store for sale is stored.

The PLU file F1 is used hereinafter as an information dictionary; however, the information dictionary may be another different file. The information dictionary stores the data for collation (feature amount) of a commodity extracted from the image data obtained by photographing the commodity. In the case where the information dictionary is another file but not the PLU file F1, the data (feature amount) stored in the information dictionary corresponds to the information (recognition information) stored in the PLU file F1. The feature amount is obtained by parameterizing external characteristics of a commodity, including the standard shape, the surface hue, the pattern and the flatness or unevenness of the commodity G.

<FIG> is a schematic diagram illustrating the data structure of the PLU file F1. As shown in <FIG>, the PLU file F1 stores the commodity information, the illustration image and the feature amount of each commodity. The commodity information includes a commodity ID which is uniquely assigned to a commodity as identification information, the commodity category the commodity belongs to, the commodity name and the unit price and other related information. The illustration image is image data which represents a commodity with a picture or illustration.

The feature amount is information representing the external characteristics, such as the tint or the surface flatness, of the commodity read from an image obtained by photographing the commodity. The feature amount is used as data for collation in the similarity degree determination process described later.

If the category of a commodity needs to be recognized (detected), information relating to commodity, such as the commodity name or unit price of a commodity, an illustration image showing the commodity and the feature amount of the commodity are managed, as shown in <FIG>. For example, if commodities belong to the type 'YY', information relating to a commodity, the illustration image and the feature amount of the commodity are managed for each category 'YY_1' , 'YY_2' or 'YY_3'. Further, in the case where a candidate is displayed with characters but not displayed with an illustration image, it is not needed to store the illustration image of the candidate in the PLU file F1.

Referring back to <FIG>, the CPU <NUM> of the POS terminal <NUM> is connected with a Store Computer SC and a communication interface <NUM> for data communication via input/output circuits (not shown). The Store Computer SC is arranged at the back office of a store. The PLU file F1 delivered to the POS terminal <NUM> is stored in the HDD (not shown) of the store computer SC.

The CPU <NUM> of the POS terminal <NUM> is connected with a connection interface <NUM> which is capable of transmitting/receipting data to or from the commodity reading apparatus <NUM>. The connection interface <NUM> is connected with the commodity reading apparatus <NUM>. The CPU <NUM> of the POS terminal <NUM> is further connected with a printer <NUM> for printing a receipt and the like. The POS terminal <NUM> prints the content of one transaction on a receipt through the printer <NUM> under the control of the CPU <NUM>.

The commodity reading apparatus <NUM> also comprises a micro computer <NUM> which is formed by connecting a ROM <NUM> and a RAM <NUM> with a CPU <NUM> via a bus line. The programs executed by the CPU <NUM> are stored in the ROM <NUM>. The image capturing section <NUM>, the distance measurement section <NUM> and a voice output section <NUM> are connected with the CPU <NUM> via various input/output circuits (not shown). The image capturing section <NUM>, the distance measurement section <NUM> and the voice output section <NUM> operate under the control of the CPU <NUM>.

The displaying and operating unit <NUM> is connected with the commodity reading unit <NUM> and the POS terminal <NUM> via a connection interface <NUM>. The displaying and operating unit <NUM> operates under the control of the CPU <NUM> of the commodity reading unit <NUM> and the CPU <NUM> of the POS terminal <NUM>. Here, the first display device <NUM> displays various screens such as a commodity candidate screen (refer to <FIG>) described later under the control of the CPU <NUM> or CPU <NUM>. Further, the first display device <NUM> displays a job selection screen (not shown) for selecting various indicated jobs. The job selection screen includes selection menus such as, for example, 'background registration', 'exception color region registration', 'feature amount registration', ' sales registration' for instructing the execution of each processing described later.

The image capturing section <NUM>, which includes an image pickup device such as a color CCD image sensor or a color CMOS image sensor, photographs through the reading window <NUM> under the control of the CPU <NUM>. For example, the image capturing section <NUM> captures dynamic images or moving pictures at 30fps. The frame images (images) sequentially captured by the image capturing section <NUM> at a given frame rate are stored in the RAM <NUM>. Further, to photograph a moving commodity G without blur, the shutter time of the image capturing section <NUM> is preferably set to be short.

The distance measurement section <NUM> includes a distance sensor such as a distance image sensor using TOF (Time Of Flight) system or a laser range scanner. The distance measurement range of the distance measurement section <NUM> is set to be substantially equal to the image photographing range of the image capturing section <NUM>. The distance measurement section <NUM> operates under the control of the CPU <NUM> to measure the distance (depth) to an object located in the image photographing range of the image capturing section <NUM>. If the image capturing section <NUM> has functions of a distance image sensor, the image capturing section <NUM> can also be used as the distance measurement section <NUM>.

The voice output section <NUM> includes a voice circuit and a loudspeaker for generating a predetermined warning voice or sound. The voice output section <NUM> outputs a warning voice or sound under the control of the CPU <NUM>.

Further, the CPU <NUM> is connected with a connection interface <NUM> which is connected with the connection interface <NUM> of the POS terminal <NUM> to transmit/receive data to or from the POS terminal <NUM>. The CPU <NUM> transmits/receives data to or from the displaying and operating unit <NUM> via the connection interface <NUM>.

Sequentially, the functional structure of the POS terminal <NUM> realized through the execution of programs by the CPU <NUM> is described below with reference to <FIG>.

<FIG> is a block diagram illustrating the functional structure of the POS terminal <NUM>. As shown in <FIG>, by executing the programs stored in the HDD <NUM>, the CPU <NUM> of the POS terminal <NUM> functions as an image acquisition section <NUM>, a depth information acquisition section <NUM>, a region segmentation section <NUM>, a similarity degree calculation section <NUM>, a similarity degree determination section <NUM>, a commodity presentation section <NUM>, an input reception section <NUM>, an information input section <NUM> and a sales registration section <NUM> which are described below.

The image acquisition section <NUM> outputs an image capturing ON-signal to the image capturing section <NUM> to start an image capturing operation. The image acquisition section <NUM> acquires the images (frame images) that are captured by the image capturing section <NUM> and stored in the RAM <NUM> after the image capturing operation starts. The image acquisition section <NUM> acquires the images in the order that the images are sequentially stored in the RAM <NUM>.

The depth information acquisition section <NUM> acquires the measurement result of the distance measurement section <NUM> as depth information, in synchronization with the image acquisition timing by the image acquisition section <NUM>.

<FIG> is a diagram exemplifying an image acquired by the image acquisition section <NUM>. As shown in <FIG>, when the operator holds a commodity G over the reading window <NUM>, the image capturing section <NUM> captures all or part of the commodity G which enters into the image capturing range R thereof. The image acquisition section <NUM> sequentially acquires the images captured by the image capturing section <NUM>. Further, the distance measurement section <NUM> measures the distance to an object (a commodity G or a part of the hand holding the commodity G or a background) located in the image capturing range R at the time the image capturing section <NUM> captures the image. The depth information acquisition section <NUM> acquires the distance to an object in the image capturing range R measured by the distance measurement section <NUM> as depth information.

The region segmentation section <NUM> segments a region showing a photographed commodity G from an image acquired by the image acquisition section <NUM> using a well-known technology such as a pattern matching technology or an edge detection technology. Specifically, the region segmentation section <NUM> detects the boundary between a commodity G and a background by extracting a contour from an image acquired. Next, the region segmentation section <NUM> segments a part of or the whole region surrounded by the contour of the commodity G as a region where the commodity G exists. A method for segmenting a region based on the foregoing well-known technology is hereinafter referred to as a conventional method.

Moreover, depending on a variety or type, or category of commodity, there is a possibility that the surface color of the commodity is almost identical to the color of the background when such a commodity is photographed. For example, in the case where the shutter time of the image capturing section <NUM> is set to be relatively short, the background is black. In this situation, if a commodity G (e.g. eggplant, avocado or grape) having a black or a nearly black surface color is photographed, the whole image captured is substantially of the same color. In this case, the boundary between the commodity G and the background is hardly detected if using the conventional method. Even if the region of the commodity G can be segmented from the image using a conventional method, the feature amount of the commodity G may not be extracted.

Thus, if the boundary between the commodity G and the background cannot be detected, that is, an image is substantially totally presented in the same color, then the region segmentation section <NUM> segments a region where the commodity G exists from the image according to depth information.

Specifically, the region segmentation section <NUM> determines, from the image, the existence position of an object closest to the reading window <NUM> (distance measurement section <NUM>) within the objects located in the image capturing range of the image capturing section <NUM>. Moreover, if a region of a given range is determined by taking the determined position (hereinafter referred to as a pixel position) in the image as a reference, then the region segmentation section <NUM> segments the determined region as a region where the commodity G exists.

Further, the determination standard for the detection of the boundary between a commodity G and a background can be set without any limitation. For example, it may be determined that the boundary between a commodity G and a background cannot be detected if the region substantially having the same color or a single color accounts for a given proportion (e.g. above <NUM>%) of the captured image.

<FIG> and <FIG> are diagrams exemplifying the operations of the region segmentation section <NUM>. <FIG> is a diagram exemplifying an image acquired by the image acquisition section <NUM>. In <FIG>, an example in which the surface color of a commodity G is substantially identical to a background is illustrated. In this situation, the region segmentation section <NUM> determines the pixel position of the object closest to the distance measurement section <NUM> (reading window <NUM>) from an image according to the depth information acquired by the depth information acquisition section <NUM>. Here, if the object at the pixel position P1 is closest to the distance measurement section <NUM>, as shown in <FIG>, the region segmentation section <NUM> sets a region AG by taking the pixel position P1 as a reference to segment the region AG. Further, in <FIG>, an example in which a rectangular region AG is extracted by taking the pixel position P1 as the center, however, the shape or size of the extracted region AG is not limited to this.

When a commodity is read by the commodity reading apparatus <NUM>, the operator holds the commodity G over the reading window <NUM> so that the commodity G can be photographed by the image capturing section <NUM>. At this time, the commodity G protrudes towards the reading window <NUM>. Thus, among the objects located in the image capturing range of the image capturing section <NUM>, the commodity G becomes closest to the reading window <NUM> (distance measurement section <NUM>). In the embodiment, focusing on the manner of the operation described above, the pixel position of the object closest to the distance measurement section <NUM> is determined in the image and a given area surrounding the pixel position is segmented. Thus, the region where a commodity G exits can be effectively segmented even if the commodity G and the background are substantially represented in the same color.

Further, to determine a situation that no commodity G held to the reading window <NUM>, that is, absence of a commodity G in the image capturing range of the image capturing section <NUM>, a threshold value is set for the determination of a closest distance (depth). Specifically, if the distance to the object closest to the reading window <NUM> (distance measurement section <NUM>) is within the threshold value (e.g. <NUM>), the region segmentation section <NUM> determines a pixel position corresponding to the object (commodity G). On the other hand, if the distance to the object closest to the reading window <NUM> (distance measurement section <NUM>) is beyond the threshold value, the region segmentation section <NUM> determines absence of a commodity G in the image capturing range instead of determining a pixel position. In this way, the execution of an unnecessary processing is prevented in the case of the absence of a commodity G in the image capturing range, thus improving processing efficiency.

Returning to <FIG>, the similarity degree calculation section <NUM> extracts feature amount from the region segmented by the region segmentation section <NUM>, that is, from a part or all of the image of the commodity G captured by the image capturing section <NUM>. Here, the feature amount extracted by the similarity degree calculation section <NUM> represents surface conditions, such as the tint and the unevenness, of the commodity G.

Further, the similarity degree calculation section <NUM> compares the feature amount of each of the commodities registered in the PLU file F1 (hereinafter referred to as a registered commodity) with that of the commodity G to calculate the similarity degree between the commodity G and each of the registered commodities. The similarity degree calculation section <NUM> calculates the similarity degree of each registered commodity with the commodity G to recognize a registered commodity the similarity degree of which is higher than a given threshold value as a candidate for the commodity G. Here, the similarity degree represents how much similar a part of or all of the image of the commodity G is with the image of a registered commodity if the similarity degree of a registered commodity is set to <NUM>%=<NUM>. Further, a weight of the surface tint and that of the unevenness is varied to calculate the similarity degree.

The recognition of an object contained in an image is referred to as a general object recognition, which is realized by using the technologies that are illustrated in the following document:<NPL>].

In addition, the technology for recognizing an ordinary object through targeted regional image segmentation is described in the following document:
Jamie Shotton: "Semantic Texton Forests for Image Categorization and Segmentation, In URL:http://citeseerx. edu/viewdoc/download?doi=<NUM>. <NUM>&rep=rep1&type=pdf (retrieved on Aug. <NUM>, <NUM>).

Additionally, no specific limitations are given to the method for calculating the similarity degree of the image of a photographed commodity G and a registered commodity. For example, the similarity degree of the image of a photographed commodity G and each registered commodity may be calculated as an absolute evaluation or a relative evaluation.

If the similarity degree is calculated as an absolute evaluation, the image of the photographed commodity G is compared with all registered commodities one by one, and a similarity degree derived from the result of the comparison is directly adopted. Further, if the similarity degree is calculated as a relative evaluation, the sum of the similarity degrees of the photographed commodity G and the registered commodities should be <NUM> (<NUM>%).

Further, the similarity degree may be a value (similarity) representing how much similar the feature amount of a commodity G with the standard feature amount of a commodity registered in the PLU file F1. Not limited to the example, the similarity degree may also be a value representing the consistency of the feature amount of a commodity G with that of each registered commodity in the PLU file F1 or a value representing how much the feature amount of a commodity G is relevant to that of each registered commodity in the PLU file F1.

The similarity degree determination section <NUM> recognizes a registered commodity (commodity ID) the similarity degree of which is above a specific threshold value within feature amounts calculated by the similarity degree calculation section <NUM> as a candidate for the photographed commodity G (hereinafter referred to as a commodity candidate). Here, in the registered commodities the similarity degree of which is higher than the threshold value, if there is a registered commodity having a similarity degree remarkably higher than other registered commodities, then the registered commodity may automatically be determined as a determined commodity. Further, the standard for the comparison of similarity degrees can be set without any limitation.

The commodity presentation section <NUM> displays, on the first display device <NUM>, information relating to a registered commodity recognized by the similarity degree determination section <NUM> as a commodity candidate. More specifically, the commodity presentation section <NUM> reads, from the PLU file F1, the record of the registered commodity recognized as a commodity candidate and displays the read record of the registered commodity on the commodity candidate screen of the first display device <NUM>.

<FIG> is a diagram exemplifying an example of the display on a commodity candidate screen. As shown in <FIG>, the commodity candidate screen A1 displayed on the first display device <NUM> includes a captured image area A11 and a commodity candidate area A12.

The captured image area A11 is used to display the captured image acquired by the image acquisition section <NUM>. A manual registration button B1 for manually executing (registering) recognition of a commodity G a department codes or commodity list is arranged under the captured image area A11. The CPU <NUM> displays a manual registration screen (not shown) for manually executing (registering) recognition of a commodity G on the first display device <NUM> according to the operation on the manual registration button B1. A commodity specified through the manual registration screen is processed as a determined commodity.

The commodity candidate area A12 is used to display information relating to a registered commodity recognized as a commodity candidate. A display region A2 is arranged in the commodity candidate area A12. <FIG> exemplifies the arrangement of four display regions A2 in the commodity candidate area A12; however, no specific limitations are given to the number of the arranged display regions A2. The commodity presentation section <NUM> displays the commodity image, the commodity name or other commodity information (XA) of a commodity candidate in the display region A2. Further, the commodity candidate displayed in the commodity candidate area A12 can be selected through the touch panel <NUM>. The operator of the commodity reading apparatus <NUM> can select, from the commodity candidates displayed in the commodity candidate area A12, a commodity candidate corresponding to the commodity G as a determined commodity.

The input reception section <NUM> receives various input operations corresponding to the displays on the first display device <NUM> through the touch panel <NUM> or keyboard <NUM>. For example, according to an operation of selecting commodity candidates, the input reception section <NUM> receives an operation of selecting a determined commodity corresponding to the commodity G from the displayed commodity candidates.

The information input section <NUM> inputs, via the connection interface <NUM>, information corresponding to the operation content received by the input reception section <NUM> into the own apparatus. For example, the information input section <NUM> inputs the information (e.g. commodity ID or commodity name) of a determined commodity determined by the operations described above. Further, the information input section <NUM> may input a sales quantity, together with a commodity ID and the like through the touch panel <NUM> or the keyboard <NUM>.

The sales registration section <NUM> carries out the sales registration of a corresponding commodity according to the commodity ID and the sale quantity input from the information input section <NUM>. Specifically, the sales registration section <NUM> records, with reference to the PLU file F1, the notified commodity ID and corresponding commodity category, commodity name, and unit price together with the sales quantity in the sales master file.

The operations of the checkout system <NUM> are described below. <FIG> is a flowchart exemplifying a sales registration processing carried out by the POS terminal <NUM>.

The CPU <NUM> of the POS terminal <NUM> starts a sales registration processing when a 'sales registration' menu is selected from the job selection screen (not shown) displayed on the first display device <NUM>. First, the image acquisition unit <NUM> outputs an image capturing ON-signal to the image capturing section <NUM> to start an image capturing operation (Act S11). Then, the image acquisition section <NUM> acquires the image captured by the image capturing section <NUM> into the RAM <NUM> (Act S12). Further, the depth information acquisition section <NUM> acquires the measurement result of the distance measurement section <NUM> into the RAM <NUM> (Act S13).

Sequentially, the region segmentation section <NUM> detects a commodity G from the acquired image, using a conventional method. Here, if the commodity G is detected (Act S14: Yes), the region segmentation section <NUM> segments the region of the detected commodity G from the image, and then Act S19 is taken.

If no commodity G is detected using the conventional method (Act S14: No), then the region segmentation section <NUM> determines whether or not the distance to the closest object (the shortest distance) is within a threshold value according to the acquired depth information (Act S16). Here, if the shortest distance is determined to be above the threshold value (Act S16: No), the region segmentation section <NUM> determines the absence of the commodity G, and Act S12 is taken.

On the other hand, if the shortest distance is determined to be within the threshold value (Act S16: Yes), the region segmentation section <NUM> determines, from the image, the pixel position where the object at the shortest distance locates (Act S17). Next, the region segmentation section <NUM> sets a region of a given range by taking the pixel position as the center (Act S18), and then Act S15 is taken to segment the area from the image.

Next, the similarity degree calculation section <NUM> extracts a feature amount of the commodity G from the region segmented in Act S15(Act S19). Afterwards, the similarity degree calculation section <NUM> compares the feature amount extracted in Act S19 with that of each registered commodity in the PLU file F1 to calculate the similarity degree of the registered commodity with the commodity G (Act S20).

The similarity determination section <NUM> determines, according to the similarity degrees calculated in Act S20, whether or not a registered commodity the similarity degree of which is above a given threshold value exists (Act S21). Here, if there is no registered commodity the similarity degree of which is above the threshold value (Act S21: No), then Act S12 is taken again. Further, if there is a registered commodity the similarity degree of which is above the threshold value (Act S21: Yes), then the similarity determination section <NUM> recognizes the registered commodity (commodity ID) as a commodity candidate.

The commodity presentation section <NUM> displays the commodity candidate recognized in Act S21 on the first display device <NUM> (Act S22). Next, the input reception section <NUM> determines whether or not the selection on a determined commodity is received through the touch panel <NUM> or keyboard <NUM> (Act S23). Here, if the selection on a determined commodity is not received (Act S23: No), Act S12 is taken again.

Further, in Act S23, if the selection on a determined commodity is received (Act S23: Yes), then the information input section <NUM> inputs the commodity ID and other information of the commodity into the POS terminal <NUM> (Act S24), and Act S25 is taken. Here, if a sales quantity is separately input through the touch panel <NUM> or keyboard <NUM>, then in Act S24, the sales quantity is input into the POS terminal <NUM> together with information representing the determined commodity. Further, a sales quantity '<NUM>' may be output as a default value when no sales quantity is input.

Next, according to the commodity ID and the sales quantity input in Act S24, the sales registration section <NUM> reads commodity information such as commodity category and unit price from the PLU file F1 and registers the read commodity information in the sales master file (Act S25).

Sequentially, the CPU <NUM> determines whether or not the job is ended with the termination of the sales registration according to an operation instruction from the keyboard <NUM> (Act S26). If the job is continued (Act S26: No), the CPU <NUM> takes Act S12 to continue the processing. If the job is ended (Act S26: Yes), the image acquisition section <NUM> outputs an image capturing OFF-signal to the image capturing section <NUM> to end the image capturing process of the image capturing section <NUM> (Act S27), then the processing (job) is ended.

As stated above, according to the POS terminal <NUM> of the present embodiment, a region where a commodity G exists is segmented from an image captured, using depth information if the boundary between the commodity G and a background cannot be detected. Thus, the feature amount of the commodity G can be extracted more practically, and the recognition rate of the commodity G can also be enhanced.

For example, it is described in the foregoing embodiment that a PLU file F1 is arranged in the POS terminal <NUM>; however, the present invention is not limited to this. For example, the PLU file F1 may also be arranged in the commodity reading apparatus <NUM> or in an external device to which the POS terminal <NUM> and the commodity reading apparatus <NUM> are accessible.

Further, the region segmentation section <NUM> of the foregoing embodiment segments a region, first using a conventional method and if not succeeded, then using depth information. However, the present invention is not limited to this; for example, the region segmentation section <NUM> may initially segment a region according to depth information without using a conventional method.

Further, in the foregoing embodiment, the present invention is applied to the POS terminal <NUM> acting as an information processing apparatus having the functions of an image acquisition unit <NUM>, a depth information acquisition unit <NUM>, a region segmentation unit <NUM>, a similarity degree calculation unit <NUM>, a similarity degree determination unit <NUM>, a commodity presentation unit <NUM>, an input reception unit <NUM> and an information input unit <NUM>. However, the present invention is not limited to this. For example, the commodity reading apparatus <NUM> including one or all of the foregoing functional units may output a calculation result, a determination result and a determined commodity to the POS terminal <NUM>.

Further, in the foregoing embodiment, the present invention is applied to the checkout system <NUM> consisting of the POS terminal <NUM> and the commodity reading apparatus <NUM> as a store system, however, the present invention is not limited to this, the store system may be a single apparatus integrating the functions of the POS terminal <NUM> and the commodity reading apparatus <NUM>. The apparatus integrating the functions of the POS terminal <NUM> and the commodity reading apparatus <NUM> may be a self-checkout apparatus (hereinafter referred to as a self-POS) used in a store such as a supermarket.

<FIG> is an oblique view illustrating the appearance of a self-POS <NUM>, and <FIG> is a block diagram illustrating the hardware structure of the self-POS <NUM>. The components shown in <FIG> and <FIG> which are identical to those shown in <FIG> and <FIG> are denoted by the same reference numerals and not described here repeatedly. As shown in <FIG> and <FIG>, a change receiver <NUM> for receiving the notes or changes for a settlement is arranged on the main body <NUM> of the self-POS <NUM>. Further, the main body <NUM> is provided with a first display device <NUM> having a touch panel <NUM> on the surface thereof and a commodity reading section <NUM> which reads the image of a commodity so as to recognize (detect) the category of the commodity.

The first display device <NUM> is, for example, a liquid crystal display. The first display device <NUM> displays various screens including a guidance screen, a registration screen and a settlement screen. The guidance screen is a screen for providing the customer the guidance instructing how to operate the self-POS <NUM>. The registration screen is a screen displaying the commodity information read by the commodity reading section <NUM>. The settlement screen is a screen on which the total amount of commodities, a pre-paid amount, a change amount and selectable payment method modes are displayed.

The commodity reading section <NUM> reads an image indicating a commodity with the image capturing section <NUM> as the customer moves the commodity across the reading window <NUM> of the commodity reading section <NUM>.

A commodity holding table <NUM> for holding an unsettled commodity put in the basket is arranged on the right side of the main body <NUM>. A commodity holding table <NUM> for holding a settled commodity is arranged on the left side of the main body <NUM>. A bag hook <NUM> for hooking a bag in which settled commodities are accommodated and a temporary holding table <NUM> for temporarily holding a settled but not bagged commodity are arranged on the commodity holding table <NUM>. Weighing devices <NUM> and <NUM> are arranged on the commodity holding tables <NUM> and <NUM> to confirm whether or not a commodity is unchanged in weight before and after the settlement.

In a case in which the above described self-POS is applied to a store system, the self-POS <NUM> functions as an information processing apparatus.

In the foregoing embodiments, the programs executed by each apparatus of the foregoing embodiments are preinstalled in the memory medium (ROM or memory unit) of the apparatus, however, the present invention is not limited to this, the programs may be recorded in a computer-readable recording medium such as a CD-ROM, a floppy disc, a CD-R and a Digital Versatile Disk (DVD) as installable or executable files. Further, the memory medium, which is not limited to a medium independent from a computer or assembled in a system, further includes a memory medium for storing or temporarily storing the programs downloaded and transferred via an LAN or the Internet.

Claim 1:
An information processing apparatus, comprising:
a first acquisition module comprising an image capturing device for photographing a given image capturing range, a first acquisition module being configured to acquire an image from the image capturing device;
a second acquisition module comprising a distance measurement device which measures a distance to an object in the image capturing range, the second acquisition module being configured to acquire a measurement result from the distance measurement device;
a segmentation module configured to segment, in the image, a region where the object exists according to the measurement result;
an extraction module configured to extract a feature amount from the region;
a calculation module configured to compare a feature amount of each article, registered beforehand in an information dictionary, with the feature amount extracted by the extraction module to calculate the similarity degree of the photographed object and the registered article, wherein the feature amount includes surface conditions of the object, wherein the calculation of the similarity degree comprises varying a weight of the surface conditions and wherein the information dictionary is connected to the information processing apparatus; and
a recognition module configured to recognize, from the articles registered in the information dictionary, a registered article the similarity degree of which is above a threshold value as a candidate for the photographed object, characterized in that the segmentation module is configured to:
extract a contour from the image and segment a region surrounded by the contour in the image, if a boundary in color between the object and the background can be detected in the image; or
segment, in the image, a region where the object exists according to an existence position of an object closest to the image capturing device measured by the distance in the image capturing range from the distance measurement device, if a boundary in color between the object and the background cannot be detected in the image.