Patent ID: 12262109

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described in detail below with reference to the accompanying drawings showing embodiments thereof.

A first embodiment of the present invention will be described below.

FIG.1is a schematic diagram showing a configuration of a digital camera100as an image capturing apparatus according to the present embodiment.

Referring toFIG.1, the digital camera100includes a lens group10, an image sensor section11, a signal processor12, a recording processor13, a recording medium14, an operation section15, and a control arithmetic unit16.

The lens group10represents a group of lenses that can be controlled so as to realize preferable image quality when an image is photographed by the digital camera100. The lens group10includes a zoom lens, a focus lens, an image stabilization lens, a diaphragm, a neutral density (ND) filter, and so forth.

The image sensor section11receives a ray of light incident through the lens group10and performs photoelectrical conversion for converting light to an electrical signal in each of pixels on its imaging surface. Further, the image sensor section11converts the electrical signals obtained by photoelectrical conversion from analog to digital to thereby generate a digital image.

The signal processor12performs a variety of kinds of image processing on the digital image generated by the image sensor section11. The image processing mentioned here refers to a variety of kinds of correction processing for realizing high image quality. Examples of the image processing include elimination of fixed pattern noise, demosaicing processing, development processing, brightness correction processing, color correction processing, geometrical deformation processing, edge emphasizing processing, gamma correction processing, contrast correction processing, aberration correction processing, and noise reduction processing. Further, the signal processor12performs not only the above-mentioned processing operations, but also recognition processing for detecting a main object area from an image, for the purpose of lens control, such as focusing and aperture control. Details of the processing operations performed in the image sensor section11and the signal processor12, respectively, will be described hereinafter. The image on which image processing has been performed by the signal processor12is transmitted to the recording processor13.

The recording processor13performs encoding on the image received from the signal processor12and transmits the encoded image to the recording medium14.

The recording medium14may be a general-purpose recording medium which can be attached/removed to and from a general-purpose interface (not shown) included in the digital camera100or may be a storage device unremovably disposed in the digital camera100and having a fixed storage capacity. The recording medium14stores image data of encoded images transmitted thereto by writing the image data in a nonvolatile storage area.

The operation section15has receiving means for receiving an operation from a user on the digital camera100and transmitting means for transmitting an operation signal indicative of details of the received operation to the control arithmetic unit16. The receiving means may be a mechanical button or an electrostatic capacitance-type touch panel integrally formed with a display member, such as a liquid crystal. Further, the receiving means may be an external remote controller connected to a general-purpose terminal or a communication device from an external terminal, such as a smartphone wirelessly connected to the digital camera100.

The control arithmetic unit16receives an operation signal transmitted from the transmitting means of the operation section15, generates control information, and transmits the generated control information to the lens group10, the image sensor section11, the signal processor12, the recording processor13, and the recording medium14. In a case where the receiving means of the operation section15is a touch panel integrally formed with the display member, the control arithmetic unit16transmits control information for displaying an image on the display member to the operation section15.

Now, there will be described a flow of signals in the entire system of the digital camera100, which is related to an operation for performing auto focusing by a switch 1 (SW1) state, referred to hereinafter, as a preliminary stage of a still image photographing by the digital camera100. Not that in the following example, it is assumed that a release button is included in the receiving means of the operation section15.

When half-pressing of the release button is detected, the operation section15transmits half-pressing information to the control arithmetic unit16. When the half-pressing information is received from the operation section15, the control arithmetic unit16determines that the digital camera100is in the SW1. In the SW1 state, to perform an operation of focusing on an object at maximum speed, it is necessary to obtain a brightness suitable for the focusing operation. Therefore, the control arithmetic unit16calculates correction values for correcting a control position of the diaphragm, exposure time, and brightness, so as to obtain brightness suitable for the focusing operation, and transmits the associated correction values to the lens group10, the image sensor section11, and the signal processor12, respectively.

The signal processor12detects a position of a main object from each of sequentially captured images, calculates object coordinates and contrast information at coordinates around the object coordinates (hereinafter referred to as the “near-object contrast information”), and transmits the calculated information to the control arithmetic unit16.

The control arithmetic unit16determines a degree of focusing in the vicinity of the object coordinates based on the near-object contrast information transmitted from the signal processor12, generates focus control information according to the determined degree of focusing, and transmits the generated focus control information to the lens group10a plurality of times. The control arithmetic unit16performs image capturing whenever the focus lens included in the lens group10is driven according to transmission of the focus control information and causes the signal processor12to calculate the near-object contrast information for each captured image. The control arithmetic unit16determines whether or not the focus lens has reached an in-focus position based on the near-object contrast information calculated by the signal processor12. If it is determined that focus lens has reached the in-focus position, the control arithmetic unit16issues a command for stopping the focus lens to the lens group10and prepares for a shift from an operation in the SW1 state to an operation in a switch 2 (SW2) state. Further, the control arithmetic unit16generates control information for the lens group10, the image sensor section11, and the signal processor12, so as to change the brightness for the focusing operation to a brightness for still image photographing.

The description has been given of the general configuration and operation of the digital camera100. Next, the internal operations of the image sensor section11and the signal processor12will be described.

First, an image sensor section11aand a signal processor12aas the conventional components, which are arranged in the same positions as the image sensor section11and the signal processor12appearing inFIG.1, will be described with reference toFIG.2.FIG.2shows a data flow inside the conventional image sensor section11aand signal processor12a.

The image sensor section11aincludes an image capturing section20and an interface (IF) section21. Further, the signal processor12aincludes an IF section22, a main image processor23, a recognition image processor24, and a recognition section25.

The image capturing section20(image capturing unit) in the image sensor section11aphotoelectrically converts received light to electrical signals and further converts the electrical signals from analog to digital to generate a digital image. The digital image output from the image capturing section20is transmitted to the IF section22in the signal processor12avia the IF section21. Here, the IF sections21and22may use general communication standards, such as low voltage differential signaling (LVDS) or sub LVDS, or may use any other special communication standards specific to the component elements. Further, althoughFIG.2shows an example in which communication between the IF sections21and22is performed via one signal line, image signals may be communicated in parallel at high speed via a plurality of signal lines.

The main image processor23in the signal processor12aperforms image processing on the image output from the image sensor section11ato generate an image to be output to the recording processor13. The image processing mentioned here includes a variety of kinds of image processing for converting an image in the Bayer array to an image which can be generally recorded and viewed. For example, the image processing includes processing for eliminating fixed pattern noise, demosaicing processing, color correction processing for adjusting the RGB balance, and gamma correction processing adapted to a gamma characteristic of a display device. Further, an object image captured through the lens group10is sometimes degraded in image quality due to characteristics of the lenses. In general, examples of the degradation include distortion aberration in which a peripheral portion is distorted, chromatic aberration in which a color shift occurs in a radial direction, and decrease in marginal illumination due to lens vignetting. The image processing mentioned here also includes processing for correcting these degradations of image quality according to the lens status when performing photographing. The distortion aberration can be corrected by performing geometrical deformation, while the chromatic aberration can be corrected by restoring each pixel by a color shift amount, and the decrease in marginal illumination can be corrected by amplifying an image signal in a concentric direction. Further, the image processing mentioned here can also include correction processing for emphasizing object edge, noise reduction processing for reducing random noise, and so forth, so as to improve the quality of the image. The image subjected to these image processing operations is output to the recording processor13arranged at a latter stage.

On the other hand, in order as to generate an image to be output to the recognition section25that performs recognition processing, the recognition image processor24(image modification unit) arranged in parallel with the main image processor23performs image processing operations similar to those performed by the main image processor23on the image output from the image sensor section11ato modify the image. However, in an object or scene to be recognized by the recognition processing, there is sometimes a brightness or gradation which makes it easy to perform recognition. For example, a black animal or the like tends to be increased in recognition accuracy by correcting the brightness to some extent, but on the other hand, a face of a person or the like tends to be lowered in a recognition rate if the light-dark contrast is low. For this reason, it is preferable that the recognition image processor24performs image processing operations different from those performed by the main image processor23according to a recognition target. The image processed by the recognition image processor24is input to the recognition section25.

In the recognition section25(second recognition unit), a variety of kinds of recognition processing are performed. The recognition processing in the recognition section25may use a function of rule-based recognition, or a function of cascade recognition for sequentially recognizing a recognition target by weak discriminators connected in series (cascade-connected), or a function of performing recognition trained for a discrimination boundary in a feature space by machine learning. Further, the recognition processing in the recognition section25may use a function (of a second learning model) of discrimination using a neural network that has obtained coefficients of pooling layers by deep learning. In a case where the recognition section25performs object recognition, as a recognition target, there may be mentioned, by way of example, a specific object, such as a person, an animal, an artificial object, the sky, a road, or a signal, and an organ as part of the object, such as a hand, a leg, a skeletal outline, a head, or a pupil. Further, the recognition section25sometimes performs scene recognition for determining a type of scene in a captured image. Examples of a scene recognized by scene recognition include specific scenes which are frequently used, such as a day scene and a night scene, an indoor scene and an outdoor scene, sunset glow, a sports scene, and a portrait. Further, recently, there is an increase in cases where the recognition section25performs, as recognition processing, class classification of properties of an object, e.g. by determining whether an object is a person or an animal, whether an object is a male or a female, and whether a object is a child or an adult. This class classification also includes image classification for determining a type into which a main object in an image is classified, such as a person, an animal, a scene, a road, the sky, or a vehicle. As a result of these recognition operations, the recognition section25outputs a position of an object (coordinates within an image), presence/absence of an object, an identifier (ID) of a determined scene, an ID of a class of the object, and an ID of an image type, to the control arithmetic unit16.

The digital camera100according to the present embodiment, which can switch the signal processor12between a recognition mode and a learning mode, will be described. More specifically, in the recognition mode, the digital camera100is capable of obtaining a recognition result using the recognition function, whereas in the learning mode, the digital camera100is capable of updating the recognition function (of a learning model) of the image sensor section11.

First, the recognition mode will be described using a data flow inside the image sensor section11and the signal processor12appearing inFIG.3. Note that the same internal components as those of the conventional image sensor section11aand signal processor12ainFIG.2are denoted by the same reference numerals, and redundant description is omitted. That is, out of the components shown inFIG.3, description of the image capturing section20, the IF section21, the IF section22, the main image processor23, the recognition image processor24, and the recognition section25, denoted by the same reference numerals as those inFIG.2, is omitted.

As shown inFIG.3, the image sensor section11is further provided with a sensor image processor31and a sensor recognition section33. That is, in the present embodiment, the digital camera100has two recognition sections, i.e. the sensor recognition section33in the image sensor section11and the recognition section25in the signal processor12.

A difference between the sensor recognition section33and the recognition section25will be described with reference toFIG.4. The sensor recognition section33(first recognition unit) as the recognition section disposed in the image sensor section11is simpler in image processing performed immediately before recognition processing, and is smaller in the scale of a circuit for recognition than the recognition section25as the recognition section disposed in the signal processor12. Therefore, the sensor recognition section33is lower in recognition performance than the recognition section25. On the other hand, in the image sensor section11, it is possible to perform recognition by first reading out only lines necessary for recognition, and output a result of the recognition in the middle of the image, and hence the sensor recognition section33can perform recognition using a thinned image or a partial image without using the whole image. Therefore, time taken to obtain a recognition result by the sensor recognition section33is shorter than time taken to obtain a recognition result by the recognition section25. Further, it is possible to output the recognition result obtained by the sensor recognition section33simultaneously with the whole image output from the image sensor section11to the outside without delay. Further, the sensor recognition section33can perform recognition not using the whole image but using a partial image differently from the recognition section25, and perform recognition processing using the circuit having a scale smaller than that of the recognition section25, and hence it is possible to make power consumption smaller than the recognition section25. Further, the recognition section25and the sensor recognition section33have their respective circuits disposed at different locations, i.e. on the image sensor section11and the signal processor12, and there is a characteristic difference in portions where heat is generated when recognition processing is performed. When the image sensor section11and the signal processor12each separately have a recognition section, it is possible to make proper selective use of the respective recognition sections by making use of the characteristic features of them. For example, it is possible to provide a parallel recognition mode in which the sensor recognition section33and the recognition section25are simultaneously used in parallel by setting different recognition targets for the sensor recognition section33and the recognition section25, respectively. Further, the recognition section25may be used when importance is placed on the recognition performance, and the sensor recognition section33may be used when importance is placed on lag of the recognition result. The recognition section25of the signal processor12may be used when it is desired to suppress heat generation in the image sensor section11, and the sensor recognition section33may be used when it is desired to suppress power consumption of the whole digital camera100. Thus, a variety of methods are envisaged for the proper use of the recognition section25and the sensor recognition section33, but in the present embodiment, the method is not limited to a specific one.

Referring again toFIG.3, when an image signal output from the image capturing section20is input to the sensor image processor31, the sensor image processor31performs image processing for converting the received image signal to an image in a format which enables the sensor recognition section33to operate the recognition function. More specifically, the image output from the image capturing section20is a RAW image, and hence the sensor image processor31performs image processing for converting the RAW image to a YUV image and performing gamma conversion. Although the image processing performed by the sensor image processor31is basically the same as the image processing performed by the recognition image processor24in the signal processor12, the scale of a circuit which can be disposed in the image sensor section11is limited, and hence the image processing performed by the sensor image processor32is limited to image processing simpler than that of the recognition image processor24. Then, the image output from the sensor image processor31is input to the sensor recognition section33. The sensor recognition section33performs the recognition processing based on a learning model32(first learning model) placed in a memory (not shown) disposed in the image sensor section11. The recognition processing performed by the sensor recognition section32is similar to that performed by the recognition section25of the signal processor12and is not particularly limited. Further, the memory storing the learning model32may be a nonvolatile memory, or may be a volatile memory into which data is loaded from a nonvolatile memory at the start of energization of the signal processor12and which holds the data during the energization time. The sensor recognition section33performs the recognition processing, and a recognition result is output to the control arithmetic unit16via the IF sections21and22. Although nFIG.3, the recognition result is output to the control arithmetic unit16via the signal processor12, there is no problem even when the recognition result is directly output to the control arithmetic unit16via the IF section21.

Next, the learning mode will be described using data inside the image sensor section11and the signal processor12appearing inFIG.5. As described above, the sensor image processor31cannot perform complicated image processing, i.e. processing requiring a large number of taps, such as geometrical deformation for correcting distortion aberration, because of its circuit scale. On the other hand, even when the circuit scale of the image sensor section11is increased, in the geometrical deformation, a corrected image cannot be generated unless pixels of the image, corresponding to a required number of taps, are read out, and hence a time lag is generated between inputting of the image to the sensor image processor31and outputting of the image from the sensor image processor31. Therefore, when it is desired to use a recognition result obtained by the sensor recognition section33without a time lag, it is impossible to perform geometrical deformation generating the above-mentioned time lag by the sensor image processor31. On the other hand, if the sensor recognition section33performs recognition on an image which has not been subjected to geometrical deformation by the sensor image processor31and remains distorted, the recognition accuracy is lowered. For example, in a case where an object is recognized from an image, it is general that a form of the object is extracted as a feature and is learned by the learning model32. However, in a case where an image has large distortion aberration caused by the lens group10, the form of the object is broken as the image extends closer to the periphery, and the image of the broken form is input to the sensor recognition section33, so that the recognition accuracy of the sensor recognition section33is lowered. On the other hand, in the signal processor12, an image on which geometrical deformation has been performed by the recognition image processor24with high accuracy is generated and input to the recognition section25. That is, the recognition section25can perform recognition processing based on the image which preserves the form of an actual object even in the periphery. In the present embodiment, the learning model32of the image sensor section11is relearned by using a result obtained by performing recognition processing by the recognition section25of the signal processor12with high accuracy, as correct answer data. This makes it possible to improve the recognition accuracy when the sensor recognition section33is used in the recognition mode.

The specific internal operations of the image sensor section11and the signal processor12in the learning mode will be described with reference toFIG.5. Note that in the learning mode, the signal processor12further includes a recognition result correction section41in addition to the components in the recognition mode, appearing inFIG.3.

A recognition result obtained by the recognition section25is returned to the image sensor section11via the recognition result correction section41. Since geometrical deformation processing for correcting distortion is not performed on the image by the sensor image processor31as described above, the sensor recognition section33performs recognition processing directly using the image having large distortion aberration caused by the lens group10as it is. To cope with this, the recognition result correction section41(correction unit) performs correction for restoring, out of the recognition results obtained by the recognition image processor24, a recognition result associated with each position within an image, to a state before the geometrical deformation. That is, the recognition result correction section41performs inverse conversion of geometrical deformation and outputs the processed recognition result to a sensor learning section43via the IF sections22and21. This makes it possible to obtain a recognition result by applying the recognition result obtained from the image geometrically deformed, to the image before being geometrically deformed.

Next, the processing performed in the image sensor section11will be described. Image processing not including geometrical deformation is performed by the sensor image processor31, and the processed image is stored in a frame memory42. The frame memory42(synchronization unit) is used to input an image output from the sensor image processor31to the sensor learning section43in a state synchronized and associated with an image used by the recognition section25to obtain the recognition result, such that these images are relevant to each other. Therefore, the frame memory42stores (holds) images of a plurality of frames sequentially input thereto which are formed by performing image processing on the images output from the image capturing section20by the sensor image processor31. Here, the geometrical deformation processing needs to read out a number of pixels of the image, corresponding to a number of taps necessary for the processing, and hence a time lag is generated between image inputting and image outputting as described above, and the recognition image processor24that outputs an image to be used by the recognition section25performs the geometrical deformation processing. Therefore, a lag of several frames (frame lag) is generated in an output of a recognition result which is output from the recognition section25and corrected by the recognition result correction41with respect to an output of an image from the sensor image processor31. As a result, if the sensor learning section43performs learning on an image directly input from the sensor recognition section31using a recognition result corrected by the recognition result correction section41, there arises a problem that the input image and the corrected recognition result do not correspond to each other. If it is assumed, for example, to perform a learning operation for continuously photographing with a fixed angle of view, there is no problem even when a lag of some frames is generated, but in a case where learning gradually progresses while the digital camera100is being normally used, i.e. while photographing is being performed while changing the angle of view, the lag brings about a serious problem. To prevent this, in the present embodiment, the sensor learning section43reads out an image of a frame input to the frame memory42at a timing earlier than the current frame by the number of frames corresponding to the time lag, from the frame memory42. Note that here, the current frame refers to a frame which is currently input from the sensor image processor31to the frame memory42. This makes it possible to prevent generation of the above-mentioned problem caused by the frame lag and solve mismatch between an image input to the sensor learning section43and a recognition result corrected by the recognition result correction section41and used for learning of the sensor learning section43. Note that the above-mentioned frame lag can be generated not only by the geometrical deformation processing performed by the recognition image processor24, but also by other processing performed in the signal processor12, and hence a total number of lag frames generated by the entire processing operations performed in the signal processor12is taken into account. With this, when the learning operation is performed, even in a case where photographing is not continuously performed with a fixed angle of view, but the digital camera100is normally used, it is possible to cause the learning to progress without any problems. The sensor learning section43(learning unit) performs learning on an image read out from the frame memory42using a recognition result input from the recognition result correction section41as teacher data. Therefore, it is possible to update the learning model32such that the learning model32achieves a recognition accuracy of the same level as a recognition accuracy achieved on an image subjected to geometrical deformation processing.

Note that the sensor learning section43can operate as part of the sensor recognition section33or can be realized as a totally separate circuit from the sensor recognition section33. Further, as the learning method, any other method may be employed insofar as it is a method making it possible to update the learning model32which can be used by the sensor recognition section33. For example, a method of updating weights of pooling layers of a neural network using e.g. a maximum likelihood estimation method, a k-means clustering method, or an evaluation function may be used. By thus using the learning mode, it is possible to improve the recognition performance of an image captured by the image sensor section11while normally using the digital camera100.

Next, a mode switching process for switching between the recognition mode and the learning mode will be described with reference toFIG.6.

FIG.6is a flowchart of the mode switching process according to the present embodiment. The present process is executed by the control arithmetic unit16(switching unit) that loads a program stored in a ROM (not shown) disposed in the digital camera100into a RAM (not shown) similarly disposed in the digital camera100. The present process is started when the digital camera100is started up.

First, when the camera is started, the operation is started in the recognition mode (step S600). The control arithmetic unit16determines, while causing the digital camera100to operate in the recognition mode, whether or not there is any object as a recognition target of the sensor recognition section33and the recognition section25, in a sequentially captured image (step S601). If there is no object as a recognition target (NO to the step S601), the process returns to the step S600, and the recognition mode is continued. On the other hand, if there is an object as a recognition target (YES to the step S601), the process proceeds to a step S602.

In the step S602, the control arithmetic unit16determines whether or not a recognition result obtained by the sensor recognition section33and a recognition result obtained by the recognition section25of the signal processor12match each other. So long as the recognition result obtained by the sensor recognition section33and the recognition result obtained by the recognition section25of the signal processor12match each other (YES to the step S602), the process returns to the step S600, and the recognition mode is continued. On the other hand, if there is a mismatch between the recognition results (NO to the step S602), to improve the recognition accuracy in the image sensor section11, the image sensor section11is switched to the learning mode and the operation in the learning mode is started (step S603).

After that, the sensor learning section43repeats learning in the learning mode, and the control arithmetic unit16determines whether or not learning satisfying a predetermined condition has ended (step S604). In the present embodiment, specifically, if the sensor learning section43has performed learning a predetermined number of times, it is determined that the learning satisfying the predetermined condition has ended (YES to the step S604), so that the process returns to the step S600to continue the operation in the recognition mode is resumed (step S604). On the other hand, if it is determined that the learning satisfying the predetermined condition has not ended (NO to the step S604), the process returns to the step S603to continue the learning mode.

In the present embodiment, as an example of the determination in the step S604, in a case where learning has been performed the predetermined number of times, it is determined that the learning satisfying the predetermined condition has ended, but this is not limitative. For example, the mode may be shifted to the recognition mode at a predetermined frequency during intervals of learning to check the recognition accuracy with respect to a recognition target, whereby when a predetermined recognition accuracy is acquired, it may be determined that the learning satisfying the predetermined condition has ended.

Further, although the determination on match/mismatch between the recognition results in the step S602is performed on one frame image in the present embodiment, this is not limitative. For example, the determination in the step S602may be sequentially performed on frame images, and in a case where a mismatch is detected a predetermined number of times, or a case where a ratio of a mismatch becomes not lower than a predetermined value, it may be determined that a mismatch has occurred between the recognition results.

Further, although the mode switching process in which the digital camera100is automatically switched between the recognition mode and the learning mode has been described with reference toFIG.6, this is not limitative. For example, if it is determined that the answer to the question of the step S602is negative (NO), a user interface screen shown inFIG.7for prompting a user to shift the mode to the learning mode may be displayed on the display section integrally formed with the operation section15, whereby the recognition mode may be shifted to the learning mode when the user selects “Yes” on the user interface screen shown inFIG.7on the operation section15.

Next, a second embodiment of the present invention will be described. There is no difference between the internal configuration of a digital camera according to the present embodiment and that in the first embodiment, and hence the same components as those described with reference toFIGS.1to6are denoted by the same reference numerals, and redundant description is omitted.

In the present embodiment, a description will be given of a case where the storage capacity is limited due to the limited circuit scale of the image sensor section11so that a lot of learning models cannot be stored. That is, although the image sensor section11can store several learning models, the number of objects as recognition targets is less than that of the recognition section of the signal processor12, and as an extreme example, there is a case where only one type of the learning model can be stored. In a case where an object desired to be recognized does not exist in the learning model32in the image sensor section11, it is necessary to operate the digital camera100in the learning mode and build the learning model32from the beginning for the object desired to be recognized by using recognition results obtained by the recognition section25as teacher data. Note that the operations in the image sensor section11and the signal processor12performed when operated in the learning mode are the same as those in the first embodiment, and hence detailed description thereof is omitted.

A mode switching process for switching between the recognition mode and the learning mode, according to the present embodiment, which is different from that of the first embodiment described with reference toFIG.6, will be described with reference toFIG.8.

FIG.8is a flowchart of the mode switching process according to the present embodiment. The present process is executed by the control arithmetic unit16(switching unit) that loads a program stored in the ROM (not shown) disposed in the digital camera100into the RAM (not shown) similarly disposed in the digital camera100. The present process is started when the digital camera100is started up.

First, when the camera is started, a user interface screen shown inFIG.9is displayed on the display section integrally formed with the operation section15to prompt a user to select an object desired to be added as a recognition target of the image sensor section11(step S800). Note that the user interface screen displayed in the step S800is not limited to the user interface screen shown inFIG.9. For example, a user interface screen that displays specific choices may be used or a user interface screen that displays an object recognized by the recognition section25from a live view image may be used.

Next, the operation of the image sensor section11is started in the learning mode to build the learning model32for recognizing the object selected in the step S800(step S801).

After that, the sensor learning section43repeats learning in the learning mode, and the control arithmetic unit16determines whether or not the learning satisfying a predetermined condition has ended (step S802). In the present embodiment, specifically, if learning has been performed on the object selected in the step S800a predetermined number of times by the sensor learning section43, it is determined that the learning satisfying the predetermined condition has ended (YES to the step S802), so that the process terminates the learning and proceeds to a step S803. On the other hand, if it is determined that the learning satisfying the predetermined condition has not ended (NO to the step S802), the process returns to the step S801to continue the learning mode.

In the step S803, the operation in the recognition mode is started.

According to the process inFIG.8, it is possible to newly add an object which is not recognized by the recognition function of the image sensor section11as a recognition target, by using the recognition function of the signal processor12. Note that the memory storing the learning model32built as described above may be a nonvolatile memory preserving stored data even after the digital camera100is powered off, or a volatile memory from which stored data is erased when the digital camera100is powered off. In the latter case, after the digital camera100is powered on, relearning of the learning model32is performed anew.

Further, in the present embodiment, as an example of the determination in the step S802, in a case where learning has been performed the predetermined number of times, it is determined that the learning satisfying the predetermined condition has ended, whereby the learning mode is shifted to the recognition mode, but this is not limitative. For example, such a user interface screen as shown inFIG.9may be displayed to prompt a user to select whether or not to shift to the recognition mode. Further, the mode may be shifted to the recognition mode when a predetermined time period elapses after the start of the learning in the step S801. Further, the learning mode may be shifted to the recognition mode at a predetermined frequency during intervals of learning to check the recognition accuracy with respect to a recognition target, whereby when a predetermined recognition accuracy is acquired, the learning mode may be shifted to the recognition mode.

As described above, although in the first and second embodiments, the digital camera100integrally formed with the lens group10has been described as the image capturing apparatus according to the present invention by way of example, this is not limitative. For example, there is no problem even when the lens group10is a separate device which can be removably attached to the body of the digital camera100. Further, the digital camera100may be implemented in another form, such as a smartphone having functions other than the camera.

The present invention has been described heretofore based on the embodiments thereof. However, the present invention is not limited to these embodiments, but it is to be understood that the invention includes a variety of forms within the scope of the gist of the invention. Further, it is possible to partially combine the embodiments on an as-needed basis.

The present invention includes a case where a program of software that realizes the functions of the above-described embodiments is supplied to a system or an apparatus having a computer that can execute the program, directly from a recording medium or using wired/wireless communication, and the system or the apparatus executes the program.

Therefore, a program code itself supplied to and installed in the computer to realize the functional processing of the present invention on the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

In this case, the program is not limited to a particular form, but insofar as it has a function of a program, it may be in any form, including an object code, a program executed by an interpreter, and script data supplied to an OS.

A recording medium for supplying the program may be e.g. a hard disk, a magnetic recording medium, such as a magnetic tape, an optical/magnetooptical storage medium, or a nonvolatile semiconductor memory.

Further, as a method of supplying the program, a method is envisaged in which the computer program implementing the present invention is stored in a server on a computer network, and a client computer connected to the server downloads and executes the computer program.

Note that in the present embodiment, the present invention can also be realized by supplying a program that realizes one or more functions to a system or a computer of an apparatus, and the system or a system controller of the apparatus performing a process for loading and executing the program. The system controller may have one or a plurality of processors or circuits, and may include a network of a plurality of separated system controllers or a plurality of separated processors or circuits, to load and execute an executable command.

The processor or circuit can include a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). Further, the processor or circuit can include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Other Embodiments

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2022-065203 filed Apr. 11, 2022, which is hereby incorporated by reference herein in its entirety.