Patent ID: 12205337

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described in detail below with reference to the accompanying drawings showing embodiments thereof.

A first embodiment of the present invention will be described below.

FIG.1is a schematic diagram showing a configuration of a digital camera100as an image capturing apparatus according to the present embodiment.

Referring toFIG.1, the digital camera100includes a lens group10, an image sensor section11, a signal processor12, a recording processor13, a recording medium14, an operation section15, and a control arithmetic unit16.

The lens group10represents a group of lenses that can be controlled so as to realize preferable image quality when an image is photographed by the digital camera100. The lens group10includes a zoom lens, a focus lens, an image stabilization lens, a diaphragm, a neutral density (ND) filter, and so forth.

The image sensor section11receives a ray of light incident through the lens group10and performs photoelectrical conversion for converting light to an electrical signal in each of pixels on its imaging surface. Further, the image sensor section11converts the electrical signals obtained by photoelectrical conversion from analog to digital to thereby generate a digital image.

The signal processor12performs a variety of kinds of image processing on the digital image generated by the image sensor section11. The image processing mentioned here refers to a variety of kinds of correction processing for realizing high image quality. Examples of the image processing include elimination of fixed pattern noise, demosaicing processing, development processing, brightness correction processing, color correction processing, geometrical deformation processing, edge emphasizing processing, gamma correction processing, contrast correction processing, aberration correction processing, and noise reduction processing. Further, the signal processor12performs not only the above-mentioned processing operations, but also recognition processing for detecting a main object area from an image, for the purpose of lens control, such as focusing and aperture control. Details of the processing operations performed in the image sensor section11and the signal processor12, respectively, will be described hereinafter. The image on which image processing has been performed by the signal processor12is transmitted to the recording processor13.

The recording processor13performs encoding on the image received from the signal processor12and transmits the encoded image to the recording medium14.

The recording medium14may be a general-purpose recording medium which can be attached/removed to and from a general-purpose interface (not shown) included in the digital camera100or may be a storage device unremovably disposed in the digital camera100and having a fixed storage capacity. The recording medium14stores image data of encoded images transmitted thereto by writing the image data in a nonvolatile storage area.

The operation section15has receiving means for receiving an operation from a user on the digital camera100and transmitting means for transmitting an operation signal indicative of details of the received operation to the control arithmetic unit16. The receiving means may be a mechanical button, or an electrostatic capacitance-type touch panel integrally formed with a display member, such as a liquid crystal. Further, the receiving means may be an external remote controller connected to a general-purpose terminal or a communication device from an external terminal, such as a smartphone wirelessly connected to the digital camera100.

The control arithmetic unit16receives an operation signal transmitted from the transmitting means of the operation section15, generates control information, and transmits the generated control information to the lens group10, the image sensor section11, the signal processor12, the recording processor13, and the recording medium14. In a case where the receiving means of the operation section15is a touch panel integrally formed with the display member, the control arithmetic unit16transmits control information for displaying an image on the display member to the operation section15.

Now, there will be described a flow of signals in the entire system of the digital camera100, which is related to an operation for performing auto focusing in a switch1(SW1) state, referred to hereinafter, as a preliminary stage of a still image photographing by the digital camera100. Not that in the following example, it is assumed that a release button is included in the receiving means of the operation section15.

When half-pressing of the release button is detected, the operation section15transmits half-pressing information to the control arithmetic unit16. When the half-pressing information is received from the operation section15, the control arithmetic unit16determines that the digital camera100is in the SW1. In the SW1state, to perform an operation of focusing on an object at maximum speed, it is necessary to obtain a brightness suitable for the focusing operation. Therefore, the control arithmetic unit16calculates correction values for correcting a control position of the diaphragm, exposure time, and brightness, so as to obtain brightness suitable for the focusing operation, and transmits the associated correction values to the lens group10, the image sensor section11, and the signal processor12, respectively.

The signal processor12detects a position of a main object from each of sequentially captured images, calculates object coordinates and contrast information at coordinates around the object coordinates (hereinafter referred to as the “near-object contrast information”), and transmits the calculated information to the control arithmetic unit16.

The control arithmetic unit16determines a degree of focusing on the object coordinates based on the near-object contrast information transmitted from the signal processor12, generates focus control information according to the determined degree of focusing, and transmits the generated focus control information to the lens group10a plurality of times. The control arithmetic unit16performs image capturing whenever the focus lens included in the lens group10is driven according to transmission of the focus control information and causes the signal processor12to calculate the near-object contrast information for each captured image. The control arithmetic unit16determines whether or not the focus lens has reached an in-focus position based on the near-object contrast information calculated by the signal processor12. If it is determined that focus lens has reached the in-focus position, the control arithmetic unit16issues a command for stopping the focus lens to the lens group10and prepares for a shift from an operation in the SW1state to an operation in a switch2(SW2) state. Further, the control arithmetic unit16generates control information for the lens group10, the image sensor section11, and the signal processor12, so as to change the brightness for the focusing operation to a brightness for still image photographing.

The description has been given of the general configuration and operation of the digital camera100. Next, the internal operations of the image sensor section11and the signal processor12will be described.

First, an image sensor section11aand a signal processor12aas the conventional components, which are arranged in the same positions as the image sensor section11and the signal processor12appearing inFIG.1, will be described with reference toFIG.2.FIG.2shows a data flow inside the conventional image sensor section11aand signal processor12a.

The image sensor section11aincludes an image capturing section20and an interface (IF) section21. Further, the signal processor12aincludes an IF section22, a main image processor23, a recognition image processor24, and a recognition section25.

The image capturing section20(image capturing unit) in the image sensor section11aphotoelectrically converts received light to electrical signals and further converts the electrical signals from analog to digital to generate a digital image. The digital image output from the image capturing section20is transmitted to the IF section22in the signal processor12avia the IF section21. Here, the IF sections21and22may use general communication standards, such as low voltage differential signaling (LVDS) or sub LVDS, or may use any other special communication standards specific to the component elements. Further, althoughFIG.2shows an example in which communication between the IF sections21and22is performed via one signal line, image signals may be communicated in parallel at high speed via a plurality of signal lines.

The main image processor23in the signal processor12aperforms image processing on the image output from the image sensor section11ato generate an image to be output to the recording processor13. The image processing mentioned here includes a variety of kinds of image processing for converting an image in the Bayer array to an image which can be generally recorded and viewed. For example, the image processing includes processing for eliminating fixed pattern noise, demosaicing processing, color correction processing for adjusting the RGB balance, and gamma correction processing adapted to a gamma characteristic of a display device. Further, an object image photographed through the lens group10is sometimes degraded in image quality due to characteristics of the lenses. In general, examples of the degradation include distortion aberration in which a peripheral portion is distorted, chromatic aberration in which a color shift occurs in a radial direction, and decreased marginal illumination due to lens vignetting. The image processing mentioned here also includes processing for correcting these degradations of image quality according to the lens status when performing photographing. The distortion aberration can be corrected by performing geometrical deformation, while the chromatic aberration can be corrected by restoring each pixel by a color shift amount, and the decreased marginal illumination can be corrected by amplifying an image signal in a concentric direction. Further, the image processing mentioned here can also include correction processing for emphasizing object edges, noise reduction processing for reducing random noise, and so forth, so as to improve the quality of the image. The image subjected to these image processing operations is output to the recording processor13arranged at a latter stage.

On the other hand, in order as to generate an image to be output to the recognition section25that performs recognition processing, the recognition image processor24(image modification unit) arranged in parallel with the main image processor23performs image processing operations similar to those performed by the main image processor23on the image output from the image sensor section11ato modify the image. However, in an object or scene to be recognized by the recognition processing, there is sometimes a brightness or gradation which makes it easy to perform recognition. For example, a black animal or the like tends to be increased in recognition accuracy by correcting the brightness to some extent, but on the other hand, a face of a person or the like tends to be lowered in a recognition rate if the light-dark contrast is low. For this reason, it is preferable that the recognition image processor24performs image processing operations different from those performed by the main image processor23according to a recognition target. The image processed by the recognition image processor24is input to the recognition section25.

In the recognition section25(second recognition unit), a variety of kinds of recognition processing are performed. The recognition processing in the recognition section25may use a function of rule-based recognition, or a function of cascade recognition for sequentially recognizing a recognition target by weak discriminators connected in series (cascade-connected), or a function of performing recognition trained by machine learning for a discrimination boundary in a feature space. Further, the recognition processing in the recognition section25may use a function of discrimination using a neural network that has obtained coefficients of pooling layers by deep learning. In a case where the recognition section25performs object recognition, as a recognition target, there may be mentioned, by way of example, a specific object, such as a person, an animal, an artificial object, the sky, a road, or a signal, and an organ as part of the object, such as a hand, a leg, a skeletal structure, a head, or a pupil. Further, the recognition section25sometimes performs scene recognition for determining a type of scene in a captured image. Examples of a scene recognized by scene recognition include specific scenes which are frequently used, such as a day scene and a night scene, an indoor scene and an outdoor scene, sunset glow, a sports scene, and a portrait. Further, recently, there is an increase in cases where the recognition section25performs, as recognition processing, class classification of properties of an object, e.g. by determining whether an object is a person or an animal, whether an object is a male or a female, and whether an object is a child or an adult. This class classification also includes image classification for determining a type into which a main object in an image is classified, such as a person, an animal, a scene, a road, the sky, or a vehicle. As a result of these recognition operations, the recognition section25outputs a position of an object (coordinates within an image), presence/absence of an object, an identifier (ID) of a determined scene, an ID of a class of the object, and an ID of an image type, to the control arithmetic unit16.

The digital camera100according to the present embodiment, which can switch the signal processor12between a recognition mode and a learning mode, will be described. More specifically, in the recognition mode, the digital camera100is capable of obtaining a recognition result using the recognition function, whereas in the learning mode, the digital camera100is capable of updating the recognition function of the signal processor.

First, the recognition mode will be described using a data flow inside the image sensor section11and the signal processor12appearing inFIG.3. Note that the same internal components as those of the conventional image sensor section11aand signal processor12ainFIG.2are denoted by the same reference numerals, and redundant description is omitted. That is, out of the components shown inFIG.3, description of the image capturing section20, the IF section21, the IF section22, the main image processor23, the recognition image processor24, and the recognition section25, denoted by the same reference numerals as those inFIG.2, is omitted.

As shown inFIG.3, the image sensor section11is further provided with a sensor image processor31and a sensor recognition section32. Further, the signal processor12is further provided with a learning model33connected to the recognition section25. That is, in the present embodiment, the digital camera100has two recognition sections, i.e. the sensor recognition section32in the image sensor section11and the recognition section25in the signal processor12.

A difference between the sensor recognition section32and the recognition section25will be described with reference toFIG.4. The sensor recognition section32(first recognition unit) as the recognition section disposed in the image sensor section11is simpler in image processing performed immediately before recognition processing, and is smaller in the scale of a circuit for recognition, than the recognition section25as the recognition section disposed in the signal processor12. Therefore, the sensor recognition section32is lower in recognition performance than the recognition section25. On the other hand, in the image sensor section11, it is possible to perform recognition by first reading out only lines necessary for recognition, and output a result of the recognition in the middle of the image, and hence the sensor recognition section32can perform recognition using a thinned image or a partial image without using the whole image. Therefore, time taken to obtain a recognition result by the sensor recognition section32is shorter than time taken to obtain a recognition result by the recognition section25. Further, it is possible to output the recognition result obtained by the sensor recognition section32simultaneously with the whole image output from the image sensor section11to the outside without delay. Further, the sensor recognition section32can perform recognition not using the whole image but using a partial image differently from the recognition section25, and perform recognition processing using the circuit having a scale smaller than that of the recognition section25, and hence it is possible to make power consumption smaller than the recognition section25. Further, the recognition section25and the sensor recognition section32have their respective circuits disposed at different locations, i.e. in the image sensor section11and the signal processor12, and hence there is also a characteristic difference in portions where heat is generated when recognition processing is performed. When the image sensor section11and the signal processor12each separately have a recognition section, it is possible to make proper selective use of the respective recognition sections by making use of the characteristic features of them. For example, it is possible to provide a parallel recognition mode in which the sensor recognition section32and the recognition section25are simultaneously used in parallel by setting different recognition targets for the sensor recognition section32and the recognition section25, respectively. Further, the recognition section25may be used when importance is placed on the recognition performance, and the sensor recognition section32may be used when importance is placed on lag of the recognition result. The recognition section25of the signal processor12may be used when it is desired to suppress heat generation in the image sensor section11, and the sensor recognition section32may be used when it is desired to suppress power consumption of the whole digital camera100. Thus, a variety of methods are envisaged for the proper selective use of the recognition section25and the sensor recognition section32, but in the present embodiment, the method is not limited to a specific one.

Referring again toFIG.3, when an image signal output from the image capturing section20is input to the sensor image processor31, the sensor image processor31performs image processing for converting the received image signal to an image in a format which enables the sensor recognition section32to operate the recognition function. More specifically, the image output from the image capturing section20is a RAW image, and hence the sensor image processor31performs image processing for converting the RAW image to a YUV image and performing gamma conversion. Although the image processing performed by the sensor image processor31is basically the same as the image processing performed by the recognition image processor24in the signal processor12, the scale of a circuit which can be disposed in the image sensor section11is limited, and hence the image processing performed by the sensor image processor32is limited to image processing simpler than that of the recognition image processor24. Then, the image output from the sensor image processor31is input to the sensor recognition section32. The sensor recognition section32performs the recognition processing. The recognition processing performed by the sensor recognition section32is similar to that performed by the recognition section25of the signal processor12and is not particularly limited. The sensor recognition section32performs the recognition processing, and a recognition result is output to the control arithmetic unit16via the IF sections21and22. Although nFIG.3, the recognition result is output to the control arithmetic unit16via the signal processor12, there is no problem even when the recognition result is directly output to the control arithmetic unit16via the IF section21.

As a difference fromFIG.2, the learning model33is provided inside the signal processor12. The recognition section25performs recognition processing using the learning model33disposed in a memory (not shown) within the signal processor12. The memory holding the learning model33may be a nonvolatile memory, or may be a volatile memory into which data is loaded from a nonvolatile memory at the start of energization of the signal processor12and is held during the energization time.

Then, the learning mode will be described referring to data in the image sensor section11and the signal processor12appearing inFIG.5.

In the learning mode, the image sensor section11has the same configuration as the configuration in the recognition mode shown inFIG.3. On the other hand, similar to the recognition mode shown inFIG.3, in the learning mode, the signal processor12has the IF section22, the main image processor23, and the learning model33. However, as shown inFIG.5, in the learning mode, the signal processor12has a learning image processor42in place of the recognition image processor24, a learning section43in place of the recognition section25, and further has a recognition result correction section40and a recognition result buffer41.

The recognition result obtained by the sensor recognition section32is input to the signal processor12via the IF sections21and22, and is input to the recognition result buffer41via the recognition result correction section40. On the other hand, an image output from the image capturing section20is input to the signal processor12via the IF sections21and22, and is input to the learning section43after being subjected to the image processing by the learning image processor42. Since distortion correction is not performed by the sensor image processor31as described above, a distorted image is used in the image sensor section11. On the other hand, the learning image processor42in the signal processor12can perform complicated image processing and can perform geometrical deformation processing for coping with distortion aberration. In other words, a recognition result obtained by the recognition section25in a case where an image from the learning image processor42is input and a recognition result obtained by the sensor recognition section32have a difference corresponding to the geometrical deformation. To cope with this, the recognition result correction section40(correction unit) performs correction for changing, out of the recognition result obtained by the sensor recognition section32, a recognition result portion associated with each position, changeable by the geometrical deformation, within an image, to a state after being subjected to the geometrical deformation. That is, the recognition result correction section40performs geometrical deformation at only predetermined coordinates, as the correction corresponding to the processing performed by the learning image processor42. This makes it possible to obtain a recognition result by applying the recognition result obtained from the image before being geometrically deformed, to the image after being geometrically deformed.

Next, the learning image processor42in the signal processor12will be described. In general, in a field of recognition engineering, as a method of improving the recognition accuracy, a generation-type learning method has been made use of. The generation-type learning method is a method of supervised learning in which discriminators that take a deteriorated image into consideration are built, by using an image formed by subjecting an image for learning (learning image) to image processing (image deterioration processing) that lowers the recognition accuracy, e.g. by adding noise on purpose, whereby a learning result is obtained with high recognition accuracy. Also in the present embodiment, the learning image processor42performs image processing for deteriorating an image on purpose. For example, the learning image processor42may perform color change processing for shifting the white balance (WB) from a proper value on purpose to thereby generate a color image lowered in visibility. The learning image processor42may perform image processing (brightness change processing) for generating an image whose brightness is shifted from a proper brightness to under-brightness or over-brightness. Further, the learning image processor42may perform image processing for deteriorating a sense of resolution by performing blur processing on an image, and image processing (contrast decrease processing) for reducing the contrast by changing allocation of gradation of an image. Further, the learning image processor42may perform image processing in a direction in which lens aberration becomes more noticeable, i.e. geometrical deformation processing for more largely distorting the periphery, aberration addition processing for changing the color of each pixel so as to make the chromatic aberration on edges more noticeable, and image processing (light amount reduction processing) for reducing the light amount on edges of the image so as to make the vignetting noticeable, and so forth. As a matter of course, assuming random noise, the learning image processor42may perform image processing (noise addition processing) for adding noise components to the whole image. The image which has been lowered in recognition accuracy by these image processing operations is output from the learning image processor42and input to the learning section43.

Next, the recognition result buffer41will be described. The recognition result buffer41(synchronization unit) is used to input a recognition result corrected by the recognition result correction section40to the learning section43in a state synchronized with an image output from the learning image processor42such that the image output from the learning image processor42is relevant to an image used by the sensor recognition section32to obtain the recognition result. That is, the learning image processor42performs complicated image processing including geometrical deformation, and hence it takes time to perform the image processing executed between inputting and outputting of an image, which generates a lag. For this reason, if the recognition result output from the sensor recognition section32, which is corrected by the recognition result correction section40, is directly input to the learning section43, the input timing of the recognition results precedes the timing of inputting an image subjected to the image processing by the learning image processor42to the learning section43. To prevent this, the recognition result buffer41holds the recognition results of images of a plurality of frames sequentially input from the recognition result correction section40in FIFO. With this, the recognition result buffer41inputs a recognition result of an image of a frame input to the recognition result buffer41at a timing earlier by the number of frames corresponding to the lag generated in the learning image processor42than timing at which the image of the current frame currently is input from the learning image processor42, to the learning section43, in synchronism with the image of the current frame. Note that here, the current frame refers to a frame subjected to processing by the learning image processor42and currently input to the learning section43. Then, the learning section43(learning unit) performs machine learning using the recognition result obtained from the recognition result buffer41as teacher data in a state synchronized and associated with the image input from the learning image processor42and stores discriminators acquired as a result of the learning in the learning model33. Here, the learning image processor42(image modification unit) may also be used as the recognition image processor24in the recognition mode, or may be configured as part of the recognition image processor24, or may be configured as totally another image processing circuit. Further, the learning section43may be a circuit also used as the recognition section25in the recognition mode or may be configured as part of the recognition section25or may be configured as totally another circuit.

Further, as the learning method, any other method may be employed insofar as it is a method making it possible to update the learning model33which can be used by the recognition section25. For example, a method of updating weights of pooling layers of a neural network using e.g. a maximum likelihood estimation method, a k-means clustering method, or an evaluation function may be used. By thus using the learning mode, it is possible to improve the recognition performance of an image captured by the image sensor section11while normally using the digital camera100.

Next, a method of switching between the recognition mode and the learning mode will be described with reference to a mode switching process inFIG.6.

FIG.6is a flowchart of the mode switching process according to the present embodiment. The present process is executed by the control arithmetic unit16(switching unit) that loads a program stored in a ROM (not shown) provided in the digital camera100into a RAM (not shown) similarly provided in the digital camera100. The present process is started when the digital camera100is started up.

First, when the camera is started up, the operation is started in the recognition mode (step S600). The control arithmetic unit16determines, while causing the digital camera100to operate in the recognition mode, whether or not an object in a sequentially captured image, which can be recognized by the sensor recognition section32, can be recognized by the recognition section25(step S601). If an object which can be recognized by the sensor recognition section32can be recognized by the recognition section25(YES to the step S601), the process returns to the step S600. On the other hand, if there is an object which cannot be recognized by the recognition section25(NO to the step S601), to add this object as an object which can be recognized by the recognition section25, the signal processor12is switched to the learning mode, and the operation in the learning mode is started (step S602). Then, it is determined whether or not a predetermined recognition accuracy is acquired with respect to the object, and if it is determined that the predetermined recognition accuracy is not acquired (NO to the step S603), the process returns to the step S602to repeat the learning in the learning mode, whereas if it is determined that the predetermined recognition accuracy is acquired (YES to the step S603), the process returns to the step S600to resume the operation in the recognition mode. Determination on whether or not the predetermined recognition accuracy is acquired will be described hereinafter.

Although in the mode switching process inFIG.6, the process in which the digital camera100automatically switches between the recognition mode and the learning mode has been described, this is not limitative. For example, a user interface (UI) screen shown inFIG.7for prompting a user to shift the mode to the learning mode may be displayed on the display section integrally formed with the operation section15, and the recognition mode and the learning mode may be switched when the user selects “YES” on the UI screen shown inFIG.7displayed on the operation section15.

The determination in the step S603on whether or not the predetermined recognition accuracy is acquired by the recognition section25of the signal processor12will be described in detail. Although this determination method is not particularly limited, for example, when the learning section43has learned input images including a predetermined number of images or more including a target object, it may be determined that the predetermined recognition accuracy is acquired. Further, as another method, after shifting to the recognition mode, recognition results obtained by the sensor recognition section32and the recognition section25, respectively, in the recognition mode, may be compared with each other, and whether or not the predetermined recognition accuracy is acquired may be determined based on a result of the comparison.FIGS.8and9each show a table in which the recognition performance is compared between the recognition section25and the sensor recognition section32. Here, a column of normal image quality indicates recognition results obtained in a case where an image photographed by the image capturing section20is directly input to the respective recognition sections. A column of under-brightness indicates recognition results obtained by the respective recognition sections in a case where an image photographed by the image capturing section20is processed by the sensor image processor31such that the brightness is reduced, and is processed by the recognition image processor24such that the brightness is reduced by the same degree. A column of over-brightness indicates recognition results obtained by the respective recognition sections in a case where an image photographed by the image capturing section20is processed by the sensor image processor31such that the brightness is increased, and is processed by the recognition image processor24such that the brightness is increased by the same degree. It is clear from the comparison inFIG.8that an under-brightness image can be correctly recognized by the sensor recognition section32(at a predetermined recognition accuracy) but cannot be correctly recognized by the recognition section25. That is, the sensor recognition section32is higher in recognition accuracy than the recognition section25, i.e. the learning has not progressed in the learning section43yet, and the recognition accuracy of the recognition section25has not reached a predetermined level. Further, it is clear from the comparison inFIG.9that not only an under-brightness image, but also an over-brightness image which cannot be correctly recognized by the sensor recognition section32can be correctly recognized by the recognition section25. That is, it is possible to determine that the recognition accuracy of the recognition section25exceeds the level of the recognition accuracy of the sensor recognition section32and has reached a predetermined level. Thus, the shift from the learning mode to the recognition mode may be determined by determining an achievement level of the recognition accuracy of the recognition section25.

Next, a second embodiment of the present invention will be described. When the learning in the learning section43has progressed, the recognition accuracy of the recognition section25exceeds the level of the recognition accuracy of the sensor recognition section32to cause a difference in performance between these two recognition sections, there occurs a case where it is difficult to make proper selective use of the recognition section25and the sensor recognition section32. To cope with this, in the present embodiment, the configuration is made, as shown inFIG.10, such that the learning model33used by the recognition section25, which has improved in performance, can also be used by the sensor recognition section32.

In the following description, the same internal components as those in the first embodiment, shown inFIG.5, are denoted by the same reference numerals, and redundant description is omitted.

As shown inFIG.10, an image sensor section11′ includes a sensor learning model60having a learning model in its internal memory area, and the sensor recognition section32performs recognition based on the sensor learning model60. Then, the sensor learning model60is connected to the learning model33in the signal processor12via the IF sections21and22, and is configured to be capable of importing discrimination parameters built in the learning model33and having reached a predetermined recognition accuracy to the sensor learning model60. Here, the recognition section25and the sensor recognition section32may use a rule-based recognition function, or a function of cascade recognition for sequentially recognizing a recognition target by weak discriminators connected in series (cascade-connected), or a function of performing recognition trained by machine learning for a discrimination boundary in a feature space. However, the recognition section25and the sensor recognition section32are required to be compatible with each other such that discrimination parameters of the learning model33can be imported to the sensor learning model60. For example, let it be assumed that the sensor recognition section32and the recognition section25include discriminators having the same configuration. In this case, by importing the discrimination parameters of the learning model33to the sensor learning model60, it is possible to improve the performance of the sensor recognition section32to a level close to the recognition section25.

As described above, although in the first and second embodiments, the digital camera100integrally formed with the lens group10has been described as the image capturing apparatus according to the present invention, by way of example, this is not limitative. For example, there is no problem even when the lens group10is a separate device which can be removably attached to the body of the digital camera100. Further, the digital camera100may be implemented in another form, such as a smartphone having functions other than the camera.

The present invention has been described heretofore based on the embodiments thereof. However, the present invention is not limited to these embodiments, but it is to be understood that the invention includes a variety of forms within the scope of the gist of the invention. Further, it is possible to partially combine the embodiments on an as-needed basis.

The present invention includes a case where a program of software that realizes the functions of the above-described embodiments is supplied to a system or an apparatus having a computer that can execute the program, directly from a recording medium or using wired/wireless communication, and the system or the apparatus executes the program.

Therefore, a program code itself supplied to and installed in the computer to realize the functional processing of the present invention on the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included int the present invention.

In this case, the program is not limited to a particular form, but insofar as it has a function of a program, it may be in any form, including an object code, a program executed by an interpreter, and script data supplied to an OS.

A recording medium for supplying the program may be e.g. a hard disk, a magnetic recording medium, such as a magnetic tape, an optical/magnetooptical storage medium, or a nonvolatile semiconductor memory.

Further, as a method of supplying the program, a method is envisaged in which the computer program implementing the present invention is stored in a server on a computer network, and a client computer connected to the server downloads and executes the computer program.

Note that in the present embodiment, the present invention can also be realized by supplying a program that realizes one or more functions to a system or a computer of an apparatus, and the system or a system controller of the apparatus performing a process for loading and executing the program. The system controller may have one or a plurality of processors or circuits, and may include a network of a plurality of separated system controllers or a plurality of separated processors or circuits, to load and execute an executable command.

The processor or circuit can include a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA). Further, the processor or circuit can include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Other Embodiments

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2022-065101 filed Apr. 11, 2022, which is hereby incorporated by reference herein in its entirety.