Patent Description:
Recently, an information processing apparatus that is able to recognize a specific recognition target from an image has been developed. As one example of the information processing apparatus, there is a character recognition apparatus that recognizes characters from an image by taking the image of an object on which the characters are written.

Here, the character recognition apparatus needs a certain amount of processing time in order to recognize the characters from the image.

However, for example, a display of the character recognition apparatus displays only the taken image during the character recognition process is being executed. Users of the character recognition apparatus cannot recognize whether the character recognition process is properly executed or not in the character recognition apparatus.

Moreover, when a plurality of character recognition results are displayed on the display, visibility of the results for users might be degraded.

This kind of character recognition apparatus is hard for users to use and is inconvenient. Therefore, for image recognition of recognition targets such as characters, development of a display that is able to enhance user convenience has been required.

<CIT> relates to techniques for detecting and recognizing text may be provided. For example, an image may be analyzed to detect and recognize text therein. The analysis may involve detecting text components in the image. For example, multiple color spaces and multiple-stage filtering may be applied to detect the text components. Further, the analysis may involve extracting text lines based on the text components. For example, global information about the text components can be analyzed to generate best-fitting text lines. The analysis may also involve pruning and splitting the text lines to generate bounding boxes around groups of text components. Text recognition may be applied to the bounding boxes to recognize text therein.

The specific aspects of the present disclosure, which have been illustrated by the accompanying drawings described above, will be described in detail below. These accompanying drawings and description are not intended to limit the scope of the present disclosure in any manner, but to explain the concept of the present disclosure to those skilled in the art via referencing specific aspects.

In a first aspect, an information processing device is provided as recited in claim <NUM>. In a second aspect, a program is provided as recited in claim <NUM>. In a third aspect, a method is provided as recited in claim <NUM>. In general, according to one example, an information processing device includes a memory and a processor.

Hereinafter, reference will now be made in detail to exemplary aspects of the disclosure are described with reference to the drawings. The implementations set forth in the following description of illustrative aspects do not represent all implementations consistent with the disclosure. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the disclosure as recited in the appended claims.

<FIG> illustrates a block diagram that shows a configuration example of information processing apparatus according to the present aspect. The information processing apparatus according to the present aspect has a function to recognize a recognition target from an image. The information processing apparatus may be a field-portable electronic device such as smartphone and tablet computer. In the following description, the information processing apparatus that recognizes characters as recognition target (hereinafter, character recognition apparatus) is mainly explained.

As illustrated in <FIG>, a character recognition apparatus <NUM> includes a camera <NUM>, storage <NUM>, a processor <NUM> and a display <NUM>.

The camera <NUM> may be embedded on the back of the character recognition apparatus <NUM> such as smartphone. The camera <NUM> takes an image of an object positioned around a user who holds the character recognition apparatus <NUM>. Here, the object includes such as advertising displays on which characters (or character strings) are written. In the present aspect, the camera <NUM> takes an image including a plurality of characters.

The storage <NUM> is implemented by a storage device (e.g., non-transitory computer-readable medium) included in the character recognition apparatus <NUM> such as HDD (Hard Disk Drive), SSD (Slid State Drive), ROM (Read Only Memory) and RAM (Random Access Memory). For example, the storage <NUM> can store the images taken by the camera <NUM>.

The processor <NUM> is implemented by a computer (for example, processing circuitry) included in the character recognition apparatus <NUM> that executes a program stored in a storage device. Here, the program executed by the computer includes operating system, application program in order to recognize a character from an image (hereinafter, character recognition application) and so on.

The processor <NUM> includes a first display processor <NUM>, a character recognition processor <NUM>, a second display processor <NUM>, a third display processor <NUM> and a search processor <NUM>. All or a part of processors <NUM> to <NUM> included in the processor <NUM> are realized by, for example, execution of the character recognition application by the computer. That is, all or a part of processors <NUM> to <NUM> are realized by software. Here, all or a part of processors <NUM> to <NUM> may be realized by a hardware such as IC (Integrated Circuit) or a combination of software and hardware. The character recognition application executed by the computer may be stored in a computer readable storage or may be downloaded by the character recognition apparatus <NUM> via network.

The first display processor <NUM> acquires the image taken by the camera <NUM>. Here, the first display processor <NUM> may acquire the image stored in the storage <NUM>, for example. Alternatively, the first display processor <NUM> may acquire the image from a server outside the character recognition apparatus. The first display processor <NUM> displays the acquired image on the display <NUM>.

The character recognition processor <NUM> executes a process for character recognition (hereinafter, character recognition process) from the image displayed by the first display processor <NUM> (the image taken by the camera <NUM>).

The second display processor <NUM> displays information that shows the character recognition process is being executed (hereinafter, feedback information) on the display <NUM> from a start time of character recognition process by the character recognition processor <NUM> to a time when the result of the character recognition process is acquired. Here, the feedback information includes information on a position of character included in the image, for example. As described above, the feedback information is displayed on the display <NUM> before the character recognition process is finished.

The third display processor <NUM> displays the result of the character recognition process on the display <NUM> when the character recognition process by the character recognition processor <NUM> is finished.

The search processor <NUM> executes search process utilizing the result of the character recognition result displayed by the third display processor <NUM>. Here, the result of the search process by the search processor <NUM> is displayed on the display <NUM>.

The display <NUM> is realized by, for example, a touch panel display included in the character recognition apparatus <NUM>.

Here, the touch panel display includes such as an electrostatic capacitive touch panel that is able to detect a contact position of a finger (or a pen) on a screen. The user is able to manipulate the character recognition apparatus <NUM> by contacting (touching) on the screen of the touch panel display with their finger.

Next, referring to a flowchart of <FIG>, one example of processing procedure by the character recognition apparatus <NUM> according to the present aspect is described.

Here, the process in <FIG> is executed by the processor <NUM>, for example, when the character recognition application in the character recognition apparatus <NUM> is activated.

First, when the character recognition application is activated in the character recognition apparatus <NUM>, the first display processor <NUM> acquires the image taken by the camera <NUM>. When the character recognition apparatus <NUM> is such as smartphone, the character recognition apparatus <NUM> has a button (for example, shutter button) for the user to give instructions to the camera <NUM> for taking the image. In this case, the first display processor <NUM> acquires the static image taken by the camera <NUM> when the user pushes the shutter button. The first display processor <NUM> displays the acquired image on the display <NUM> (step S1). Here, the image acquired by the first display processor <NUM> may be stored in the storage <NUM>. While here explains the case that the static image taken by the camera <NUM> is acquired and displayed on the display <NUM>, the first display processor <NUM> displays a movie taken by the camera <NUM> on the display <NUM>.

Next, the character recognition processor <NUM> executes the character recognition process in order to recognize characters (character strings) included in the image displayed by step S1 (step S2).

Here, in the character recognition process, the process for detecting an area that includes the character string from the image displayed by the step S1 (hereinafter, character string area) is executed. Next, the process for recognizing the character string included in the detected character string area is executed and text that represents the character sting is acquired as the result of the character recognition process (hereinafter, character recognition result).

The character recognition processor <NUM> may execute translation process for the acquired text. By this translation process, translation result that the text is translated into a predefined language (that is, translation sentence of the text) is acquired.

In other words, the character recognition result according to the present aspect may be text that represents character string included in the image (character string area) or may be translation sentence of the text.

In this aspect, the character recognition process is mainly explained as a process that includes the process for detecting character string area form the image and the process for recognizing character string included in the detected character string area (the process for acquiring the text that represents the character string). In addition, the character recognition process may also include the translation process for the text.

Here, the character recognition process needs a certain amount of time in order to acquire the character recognition result. Therefore, the second display processor <NUM> displays the feedback information that indicates the current character recognition process is being executed while the character recognition process by the character recognition processor <NUM> is executed (that is, until the character recognition result is acquired) (step S3). Here, the feedback information is information in order to feedback processing status of character recognition to the user who is using the character recognition apparatus <NUM>.

In particular, the second display processor <NUM> displays a partial area of the image as the feedback information, for example. The partial area is an area that is estimated to include characters by the character recognition process (the process for detecting character string area). Some examples of the feedback information displayed on the display <NUM> are described later.

After the execution of step S3, step S4 judges whether the character recognition process by the character recognition processor <NUM> is finished or not (step S4).

When the character recognition process is judged to be finished (YES in step S4), the character recognition processor <NUM> acquires the character recognition result. The third display <NUM> displays the character recognition result acquired by the character recognition processor <NUM> on the display <NUM> (step S5). Here, if the character recognition process includes the translation process as described above, the character recognition process is judged to be finished when the translation process is finished (when the translation sentence is acquired).

On the other hand, if the character recognition process is judged to be not finished (NO in step S4), it goes back to step S3 and continues the process. In other words, in the process of <FIG> the display of the feedback information is kept until the character recognition process is finished. Here, the feedback information may be update according to, for example, the progress of the character recognition process (processing status) (that is, the different feedback information may be displayed based on the processing status).

Hereinafter, it concretely describes transition examples of screen displayed on the display <NUM> (hereinafter, display screen) when the character recognition process is executed by the character recognition apparatus <NUM>.

As described above, when the character recognition application in the character recognition apparatus <NUM> is activated and the image is taken by the camera <NUM>, the image (including the display screen) is displayed on the display <NUM>.

<FIG> shows one example of display screen in the case that the image taken by the camera <NUM> (hereinafter, taken image) is displayed.

As shown in <FIG>, a first area <NUM> and a second area <NUM> are set to the display screen.

The first area <NUM> is an area to show the taken image (static image). In the example of <FIG>, a taken image <NUM> that includes the signboard in which the character strings of "XXX STATION" and "TURN RIGHT AT THE NEXT CORNER" are written is displayed.

The second area <NUM> is an area to show the result of character recognition process to the taken image displayed on the first area <NUM>. As shown in <FIG>, before executing the character recognition process, the second area <NUM> is blank.

Next, the character recognition process for the taken image <NUM> in <FIG> is executed. In the character recognition process, as described above, the process for detecting character string area and the process for acquiring text that represents character string in the character string area are executed.

While the character recognition process is executed, the feedback information that indicates the present character recognition process is being executed is displayed.

Hereinafter, it concretely explains the feedback information. In this aspect, the feedback information includes, for example, a likelihood map image (information on likelihood) that represents likelihood of character existence for each pixel that configures the taken image.

In this case, the second display processor <NUM> extracts a plurality of character candidate areas from the taken image (original image) and adds one to likelihood for each pixel included in the character candidate areas, for example. For example, the character candidate area is a part of the taken image and has a possibility to include characters. For example, extraction of the character candidate area is executed by extracting feature parts such as edges based on luminance value of each pixel of the taken image and specifying an area that has features of character likeness.

As described above, a plurality of character candidate areas are extracted. The likelihood for pixels that have high number of extracted times as character candidate area becomes high and the likelihood for pixels that have low number of extracted times as character candidate area becomes low. The second display processor <NUM> generates a likelihood map image by adding color according to the likelihood of each pixel to the each pixel. In this aspect, the feedback information includes color information. The above method of likelihood map generation is one example and the likelihood map may be generated by using other known technologies.

In this example, a likelihood map image generated by the second display processor <NUM> is displayed on the display <NUM>, for example, after the execution of character recognition process starts and until a character string area is detected.

<FIG> illustrates one example of display image when a likelihood map image is displayed as feedback information.

As in <FIG>, a likelihood map image <NUM> is displayed on the taken image <NUM> that is displayed on the first area <NUM>. Here, an area <NUM> in the likelihood map image <NUM> in <FIG> illustrates an area that includes low likelihood pixels (in other words, an area that has low possibility of character existence). On the other hand, an area <NUM> in the likelihood map image <NUM> illustrates an area that includes high likelihood pixels (in other words, an area that has high possibility of character existence).

In particular, in the likelihood map image <NUM>, the area <NUM> that has lower likelihood than a predefined value is displayed by black (or gray), for example. The area <NUM> that has higher likelihood than the predefined value is displayed by white.

While the area <NUM> is explained to be displayed by one color (white), the area <NUM> may be displayed by a plurality of colors according to likelihood of each pixel, for example.

In this way, the user of the character recognition apparatus <NUM> is able to understand that the character recognition apparatus <NUM> is executing character recognition process and that the area <NUM> is estimated to include characters by the character recognition apparatus <NUM> (in other words, a part of image where the character recognition image recognizes as area to include characters).

In <FIG>, it explains the case that the likelihood map <NUM> is displayed on the taken image <NUM>. For example, in one configuration, the taken image <NUM> may be hidden when the character recognition process starts and only the likelihood map image <NUM> may be displayed.

Moreover, instead of displaying the likelihood map image <NUM>, frame borders <NUM> in <FIG> that represent each of the plurality of character candidate areas (frame borders that include areas on the taken image <NUM> where there is a possibility of character existence) may be displayed on the taken image <NUM> as feedback information. The frame borders <NUM> in <FIG> may be displayed on the likelihood map image <NUM>. By displaying the frame borders <NUM>, rough size of detected character can be notified to the user. While the frame borders <NUM> in <FIG> are rectangular shape, the frame borders <NUM> may be other shapes such as circle.

In other words, in this aspect, the feedback information (information on positions of recognition targets) includes information that indicates a part of image or information on frame borders that include a part of image.

It explains the case that the character string area is detected by the character recognition processor <NUM>.

In this case, the second display processor <NUM> hides the likelihood map image <NUM> and displays frame borders that represent bounding rectangles of detected character areas (hereinafter, frame borders of character string areas) as feedback information.

<FIG> illustrates one example of display screen in the case that a frame border of character string area is displayed as feedback information.

As illustrated in <FIG>, frame borders of character string areas are displayed on the taken image <NUM>, for example. In <FIG>, the frame border <NUM> of character string area that includes the character string of "XXX STATION" and the frame border <NUM> of character string area that includes the character string of "TURN RIGHT AT THE NEXT CORNER" written on the signboard in the taken image <NUM> are displayed.

The frame borders <NUM> and <NUM> in <FIG> are displayed on the display <NUM> (the first area <NUM>) until the character recognition results are obtained by the character recognition process, for example.

In this way, the user of the character recognition apparatus <NUM> is able to understand that the character recognition apparatus <NUM> is executing character recognition process and that areas included by the frame borders <NUM> and <NUM> are detected as character string areas.

The display forms (shape, color and so on) of the frame borders <NUM> and <NUM> may be changed according to detection accuracy (likelihood) of character string areas included in each of the frame borders <NUM> and <NUM> in <FIG>. For example, the detection accuracy of character string area may be a summation of likelihood of pixels included in the character string area among likelihood of pixels included in the taken image <NUM>.

<FIG> shows the case that the frame borders <NUM> and <NUM> are displayed. Alternatively, the frame borders <NUM> and <NUM> may be displayed when detection process for all character string areas is finished. The frame borders <NUM> and <NUM> may be displayed sequentially every time when each character string area is detected.

Although the frame borders <NUM> and <NUM> in <FIG> are rectangular shape, the frame borders <NUM> and <NUM> may be other shapes such as circle.

Here, it assumes that the character recognition process is finished and the character recognition result (for example, texts that represent strings included in the character string area) is obtained by the character recognition processor <NUM>.

In this case, the third display processor <NUM> hides the frame borders of character recognition areas and displays the character recognition result obtained by the character recognition processor <NUM>.

<FIG> illustrates one example of display screen when the character recognition result is displayed. As illustrated in <FIG>, the character recognition result is displayed on the second area <NUM> in the display screen.

In the example of <FIG>, texts of "XXX STATION" and "TURN RIGHT AT THE NEXT CORNER" are displayed on the second area <NUM> as the character recognition result for the taken image <NUM>.

Here, it explains the case that texts that represent the character strings included in the character string areas are obtained as the character recognition result. When translation result (translation sentence) of the texts is obtained as the character recognition result, the translation result may be displayed on the second area <NUM>. Both texts and translation result of the texts may be displayed on the second area <NUM>.

In this aspect, the display screen includes the first area <NUM> and the second area <NUM>. Alternatively, the first area <NUM> may be set on the whole of the display screen until the character recognition result is obtained and the second area <NUM> may be set on the display screen when the character recognition result is obtained (when the character recognition result is displayed).

Here, a process to obtain texts representing character strings included in character string areas is executed sequentially for each character string area. In particular, for example, the text representing the character string included in the character string area of the frame border <NUM> is obtained after the text representing the character string included in the character string area of the frame border <NUM> is obtained.

In the example of <FIG>, the frame border <NUM> and <NUM> of character string areas are hidden when the character recognition result is obtained. Alternatively, the display of the frame border <NUM> and <NUM> of character string areas may be kept even when the character recognition result is obtained. In this case, the frame border <NUM> may be highlighted when the text representing the character string included in the character string area of the frame border <NUM> is obtained and the frame border <NUM> may be highlighted when the text representing the character string included in the character string area of the frame border <NUM> is obtained. In particular, as illustrated in <FIG>, the frame border <NUM> is highlighted when the text representing the character string of "XXX STATION" included in the character string area of the frame border <NUM> is obtained. After that, as illustrated in <FIG>, the frame border <NUM> is highlighted when the text representing the character string of "TURN RIGHT AT THE NEXT CORNER" included in the character string area of the frame border <NUM> is obtained.

The frame border may be highlighted when the character string (or text) of the frame border displayed on the second area <NUM> is specified (or selected) by user. In particular, the frame border <NUM> is highlighted when the character string of "XXX STATION" displayed on the second area <NUM> is specified. The frame border <NUM> is highlighted when the character string of "TURN RIGHT AT THE NEXT CORNER" displayed on the second area <NUM> is specified.

Alternatively, only the frame border <NUM> may be displayed (the frame border <NUM> is hidden) when the character string of "XXX STATION" displayed on the second area <NUM> is specified. Only the frame border <NUM> may be displayed (the frame border <NUM> is hidden) when the character string of "TURN RIGHT AT THE NEXT CORNER" displayed on the second area <NUM> is specified.

With the configuration described above, the user is able to understand the character string area corresponding to the character recognition result (that is, the character string area that includes the character string of the text).

When the character string area is detected and the character string included in the character string area is not a recognizable language (for example, the character string is not an order of words of recognizable language), as illustrated in <FIG>, the result that shows the character (string) is not be able to be recognized is displayed on the second area <NUM>.

Alternatively, when the frame border that includes the character string area is to be displayed, the frame border that includes unrecognizable character string may be displayed by other ways such as broken line. For example, as illustrated in <FIG>, the frame border <NUM> may be displayed by broken line when the character string included in the character string area of the frame border <NUM> is recognizable and the character string included in the character string area of the frame border <NUM> is unrecognizable. Alternatively, the frame border <NUM> may be displayed by other colors (that show the character is unrecognizable) or may be hidden.

The process illustrated in <FIG> is executed every time when an image is taken by the camera <NUM> when the user pushes the shutter button.

In this case, when a first image is taken by the camera <NUM>, the character recognition result of the first image is displayed on the second area <NUM>. After that, when a second image is taken by the camera <NUM>, the character recognition result of the second image is displayed on the second area <NUM>. Here, when the user needs to confirm the character recognition result for the first image after the second image is taken, it needs to retake the first image by the camera <NUM> and execute the character recognition process (alternatively, it needs to read the first image from the storage <NUM> and execute the character recognition process). However, it is inconvenient for the user.

In order to solve this problem, in this example which is in accordance with an embodiment, the history of the character recognition results is displayed on the second area <NUM>. In particular, when the second image is taken after taking the first image, the character recognition results for both the first image and the second image are displayed on the second are <NUM>. When the character recognition process for other images is executed, the character recognition results for the other images are added to the second area <NUM>. In this case, thumbnail image of the first image is displayed on the second area <NUM> in correspondence with the character recognition result for the first image. And, a thumbnail image of the second image is displayed on the second area <NUM> in correspondence with the character recognition result for the second image.

The first image and the second images may be a still image, in an example not encompassed by the claims. The first and the second images are images constituting a movie displayed on the display <NUM> (the first area <NUM>) in real-time.

When the character recognition process for images constituting the movie displayed in real-time on the display <NUM> is executed, the same character recognition results are obtained many times. In this case, it is not helpful for the user to display the same character recognition results in the history of the character recognition results. Therefore, in this aspect, when the same character recognition result as the past-obtained character recognition result (text) is obtained, these character recognition results are collected and displayed as one character recognition result on the second area <NUM>.

Instead of simply collecting the same character recognition results, only the character recognition results from the same character string area may be collected and displayed as one character recognition result based on information on imaging targets (objects) by matching between images or measuring locations of the character recognition apparatus <NUM> (the camera <NUM>) and so on. In particular, by taking correlations among a plurality of images of which size are normalized and judging similarities among the plurality of images, the character recognition results obtained from the similar images can be collected and displayed as one character recognition result.

In the same manner, it is possible to collect the character recognition results corresponding to the same character string area by judging identity of imaging targets based on the locations of the character recognition apparatus <NUM>, the angles of the camera <NUM> and so on.

In this aspect, when the imaging targets (a first recognition target and a second recognition target) included in each of the plurality of images are the same, one of display processes for the character recognition results of the plurality of images may be executed (for example, display process for the character recognition result of the first recognition target may be executed).

When judging the identity of imaging targets, the judging process is executed before executing the character recognition process to skip the character recognition process for the same imaging target and reduce process amount of the character recognition apparatus <NUM>.

For example, the locations of the character recognition apparatus <NUM> can be obtained from GPS function embedded in the character recognition apparatus <NUM> (for example, smartphone). The angles of the camera <NUM> can be judged by accelerator sensor embedded in the character recognition apparatus <NUM>.

When a plurality of character recognition results are displayed on the second area <NUM> and there is no space to display a new character recognition result, the past character recognition result may be deleted from the second area in the order of acquisition.

Although both the character recognition result for the image (taken image) displayed on the first area <NUM> and the history of character recognition results are displayed on the second area <NUM>, an area for displaying the character recognition result for the image displayed on the first area <NUM> and an area for displaying the history of character recognition results may be separated.

When the movie is displayed on the screen <NUM> in real-time, only the character recognition result for the movie (image displayed in real-time) may be displayed instead of displaying the history of character recognition results (the past character recognition results). It is possible to switch between displayed and hidden of the history of character recognition results based on the kind of recognition target image (still image or movie).

In this aspect, the character recognition result obtained by the character recognition process is displayed on the second area <NUM>. The character recognition result displayed on the second area <NUM> may be utilized for search process.

Hereinafter, referring to the flowchart of <FIG>, one example process of character recognition apparatus <NUM> in the case that the character recognition result is utilized for search process is described.

Here, it assumes that the character recognition result is displayed on the second area <NUM> by executing the process illustrated in <FIG>. In this case, the user of the character recognition apparatus <NUM> is able to operate the character recognition apparatus <NUM> for specifying the character recognition result displayed on the second are <NUM> (hereinafter, specifying operation). In the case that the character recognition apparatus <NUM> has the configuration of touch panel display, the specifying operation includes operation to touch the character recognition result displayed on the second area <NUM> by finger, for example.

When the specifying operation is executed by user, the search processor <NUM> included in the processor <NUM> obtains the character recognition result specified by the specifying operation (step S11). The character recognition result obtained by step S11 includes text representing character string included in the taken image.

Next, the search processor <NUM> executes search process based on the character recognition result obtained by step S11 (step S12). In this search process, for example, web pages (site) are searched via internet by using the character recognition result (text) as search word.

The result of the search process (search result) in step S12 is displayed on the screen <NUM> (step S13). In this case, for example, the search result is displayed on the second area <NUM>.

Here, it concretely explains the process illustrated in <FIG>. It assumes that the texts of "XXX STATION" and "TURN RIGHT AT THE NEXT CORNER" are displayed on the second area <NUM> as the character recognition result for the taken image <NUM>.

In this case, for example, the user is able to execute the specifying operation to specify at least one word of the plurality of words constituting the texts displayed on the second area <NUM>. For example, the user is able to touch (specify) the words "XXX STATION" as search keyword.

With this operation, the search processor <NUM> searches web pages by using the "XXX STATION" as search keyword. In this case, for example, the search results including time table (web page) of "XXX station" are displayed on the second area <NUM> in the form of thumbnail. As for the search results, some web pages that have higher scores (matching degrees) of consistence with the search keyword are displayed.

The user is able to display the time table (web page) of "XXX STATION" on the screen <NUM> (the first area <NUM> or the whole screen) by specifying the search result (thumbnail) in the second area <NUM>.

Here, in this aspect, the character recognition result may be translation sentence of text representing the character string included in the taken image. When the translation sentence is utilized to search web pages, the search process may be executed based on the translation sentence or the text before translation.

It explains the case that web pages are searched. Alternatively, for example, it may be the configuration to search applications (programs) installed in the character recognition apparatus <NUM>. For example, when "XXX STATION" is specified as search keyword, the application used for train transit guidance may be searched and started automatically. Alternatively, the application to show a map (map information) may be searched and the map around the "XXX STATION" may be displayed automatically.

Instead of the application, for example, it may be the configuration to search various kinds of contents stored in the storage <NUM>. When images taken by the camera <NUM> are stored in the storage <NUM>, the image from which specified character recognition result is obtained may be searched from the storage <NUM> and displayed on the screen <NUM> (for example, in the first area <NUM>). In this case, the image and the character recognition result of the image may be linked in the character recognition apparatus <NUM> (the storage <NUM>).

By storing location information that shows the location where the image is taken in the storage <NUM> and correlating the location information with the image, the image of the character recognition result and the location where the image is taken can be displayed when the character recognition result is specified. With this configuration, in the case that a plurality of images that include names of restaurants are taken, the image of the signboard that shows the name of restaurant and the location of the signboard (the location where the image is taken) can be known.

Instead of searching images from the storage <NUM>, various contents stored in the outside server may be searched.

When the search result is displayed on the second area <NUM>, there might not be enough space to show the search result in the second area <NUM> illustrated in <FIG>. Therefore, when the search process is executed (that is, when the search result is displayed on the second area <NUM>), the area of the second area <NUM> may be widened. In this case, although the first area <NUM> becomes small, the display area of the taken image <NUM> in the first area <NUM> may be adjusted to include at least the character string area (string included in the character string area).

In this aspect, the character recognition process that includes the process for detecting character string area from the image (a first process) and the process for recognizing characters included in the detected area (a second process) are executed, and the feedback information that shows the character recognition process is being executed is displayed while the character recognition process is executed, and the character recognition result is displayed when the character recognition process is finished. In this aspect, an area of image that is estimated to include characters in the process of detecting character string area is displayed as the feedback information.

In particular, the likelihood map image that has color according to the likelihood (degree to include characters) of each pixel or the frame borders that include the areas that have possibility to include the characters are displayed as the feedback information.

This aspect is able to show the processing status of character recognition to the user and the user is able to understand that the character recognition process for the image is ongoing and the user does not feel that waiting time is long. In other words, this aspect is able to achieve the display for character recognition in order to enhance the utility for user.

In this aspect, it is possible to show the area (or its location) that the character recognition apparatus <NUM> recognizes as character location to user and the user is able to know whether the character recognition process is being executed appropriately (that is, whether the character intended by the user is recognized). In particular, for example, when the likelihood map image <NUM> in <FIG> is displayed, the user is able to know that character locations in the taken image <NUM> are recognized appropriately. On the other hand, when the likelihood map image is displayed on the whole area of the area <NUM>, the user is able to know that character locations are not recognized appropriately. In this case, the user is able to retake the image or execute the character recognition process again.

In this aspect, one example of feedback information is explained. The feedback information is information in order to show that the character recognition process is being executed and the feedback information may be a different configuration in the course of the character recognition process. In particular, as illustrated in <FIG>, the recognition target area <NUM> (the area that has high possibility to include characters) may be displayed by mosaic pattern. Alternatively, as illustrated in <FIG>, the specific character strings (information on character strings that show the character recognition process is in progressing) such as "IN PROGRESSING" may be displayed on the area <NUM>. Alternatively, the change of images may be utilized in order to show that the character recognition process is being executed (not shown in figure).

In this aspect, the image is displayed on the first area <NUM> of the character recognition apparatus <NUM> (the display <NUM>) and the character recognition result (the result of character recognition process) is displayed on the second area <NUM> of the character recognition apparatus <NUM> (the display <NUM>). By taking this configuration of this aspect, user's visibility for the character recognition result is improved.

While the second area <NUM> is positioned below the first area <NUM> in this aspect, the second area <NUM> may be placed above the first area <NUM> or the second area <NUM> may be placed left side or right side of the first area <NUM>. Alternatively, the position of the second area <NUM> may be changed based on the character (character string area) included in the image displayed on the first area <NUM>. In particular, the second area <NUM> may be placed near the character string area.

While the area to show the image (the first area <NUM>) and the area to show the character recognition result (the second area <NUM>) are separately placed, the image is displayed on the whole screen and the character recognition result may be displayed on the character (string) included in the image.

Moreover, in this aspect, the character recognition processes for the plurality of images (the first image and the second image) are executed and the plurality of character recognition results (the results of the first character recognition process and the second character recognition process) may be displayed as history. By taking the configuration of this aspect, when the character recognition result for the past image taken by the camera <NUM> is needed, the user is able to confirm the character recognition result without retaking the image.

In this aspect, when the plurality of character recognition results displayed as history are the same, the plurality of character recognition results are collected and displayed (that is, any one of the plurality of character recognition results is displayed). By taking this configuration, the area to show the history (the second area <NUM>) is utilized efficiently.

In this aspect, when the character recognition result displayed on the second area <NUM> is specified by user, the image from which the character recognition result is obtained is displayed. By taking this configuration, the user is easily able to confirm the image from which the character recognition result is obtained even when the image was taken in the past by specifying the character recognition result.

In this aspect, when the character recognition result displayed on the second area <NUM> is specified by user, search process for the character recognition result (text) is executed and the search result is displayed. By taking the configuration of this aspect, the user is easily able to search information on the character recognition result and the usability for the user is improved.

The character recognition apparatus <NUM> described in this aspect may be a configuration that includes at least one of display examples described in this aspect or may be a configuration that includes a combination of the display examples.

In this aspect, for example, the camera <NUM> takes images that include signboards in the city. The objects taken by the camera <NUM> may be anything that includes characters. The objects may be menu, guidance board, merchandise tag, book, and so on. The objects may be display that shows characters.

While the character recognition apparatus <NUM> according to this aspect is a portable device such as smartphone or tablet computer as one example, the character recognition apparatus <NUM> may be implemented by a server that connects to an electronic device that includes the camera <NUM>.

While the character recognition apparatus is described to recognize characters from the image in this aspect, the aspect may be applied to the case where objects other than characters are recognized from the image. In particular, the aspect may be applied to the information processing apparatus (recognition apparatus) that recognizes face, body, sign (mark), mobile object, and so on. When recognizing the objects, any know technology that utilizes statistical training can be utilized.

Claim 1:
An information processing device (<NUM>) comprising:
a memory (<NUM>);
a camera (<NUM>);
a display (<NUM>); and
a processor (<NUM>) configured to:
set a first area (<NUM>) and a second area (<NUM>) on the display screen, wherein the first area (<NUM>) is an area to show a taken image and the second area (<NUM>) is an area to show the character recognition result corresponding the taken image displayed on the first area (<NUM>);
recognize a first target included in a first image (<NUM>);
display first feedback information indicating that the recognition for the first recognition target is being executed before the recognition is ended, and display a character recognition result of the first recognition target after the recognition has been ended;
recognize a second recognition target included in a second image (<NUM>) different from the first target; and
display second feedback information indicating that the recognition for the second recognition target is being executed before the recognition is ended, and display a character recognition results of the second recognition target after the recognition has been ended,
and characterized in that
the character recognition result of the first recognition target and a thumbnail image of the first image (<NUM>) and the character recognition result of the second recognition target and a thumbnail image of the second image (<NUM>) are displayed as a history of the character recognition result, such that, when the second image is taken after taking the first image, the character recognition results for both the first image and the second image are displayed,
wherein the first and the second images are images constituting a movie taken by the camera and displayed on the first area (<NUM>) in real-time, and
wherein when the first recognition target of the first image (<NUM>) and the second recognition target of the second image (<NUM>) are the same, the recognition of the second recognition target is not performed.