Patent Description:
In the related art, there is known a technique of capturing light of a light pen or the like and detecting the light from the obtained image. For example, <CIT> describes a pointer light tracking method in which pointer light is captured by a camera and the pointer light on a display is tracked by a computer based on the obtained image.

In the technique described in <CIT>, detection of the pointer light is facilitated by adjusting a camera shutter speed or the like to acquire an image in which the pointer light is projected strongly.

In the technique described in <CIT>, in an image in which the pointer light is projected strongly, the pointer light can be easily detected, but the image quality of the background region may deteriorate. <CIT>, <CIT> and <CIT> relate to light painting photography in a dark environment consisting of capturing trajectories of a moving light source within a long exposure time period, extracting light trajectories to form a light image and generating a drawing image constituted by combining the light image with a background image.

The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to provide an image generation device, an image generation method, and a non-transitory storage medium storing an image generation program capable of obtaining a high quality drawing image even in a background region.

The present invention is defined in independent claims <NUM>, <NUM> and <NUM> as appended. Further preferable embodiments are given by the dependent claims.

According to the present disclosure, it is possible to obtain a high quality drawing image even in a background region.

Hereinafter, examples of embodiments for performing the technique of the present disclosure will be described in detail with reference to the drawings.

A configuration of a space drawing system <NUM> of the present embodiment will be described with reference to <FIG>. As shown in <FIG>, the space drawing system <NUM> of the present embodiment includes a drawing device <NUM> and a smartphone <NUM>. The drawing device <NUM> and the smartphone <NUM> are connected to each other via a network <NUM> by wired communication or wireless communication.

A configuration of the drawing device <NUM> according to the present embodiment will be described with reference to <FIG>. The drawing device <NUM> of the present embodiment is a device for a user to perform drawing in space with light emitted from a light source <NUM> (see light <NUM> in <FIG>) in a case where the user, who performs the drawing, holds and moves the drawing device <NUM> that emits light from the light source <NUM>. As an example, in the present embodiment, a light pen type drawing device <NUM> will be described. The drawing device <NUM> of the present embodiment is an example of an instruction device of the present disclosure.

On the other hand, the smartphone <NUM> includes a camera <NUM> that captures light emitted from the light source <NUM> of the drawing device <NUM>. The smartphone <NUM> has a function of detecting the light from the light source <NUM> of the drawing device <NUM> from the captured image captured by the camera <NUM> and generating a drawing image based on a detection status. The smartphone <NUM> of the present embodiment is an example of an image generation device of the present disclosure.

As an example, in the present embodiment, a user, who performs drawing by using the drawing device <NUM>, and a user, who captures light emitted from the drawing device <NUM> to generate a drawing image by using the smartphone <NUM>, are different users. In the present embodiment, in order to distinguish between the two users, a user, who performs drawing by using the drawing device <NUM>, is referred to as a "drawing user", and a user, who performs imaging and generation of a drawing image by using the smartphone <NUM>, is referred to as an "imaging user".

First, the details of the drawing device <NUM> will be described. <FIG> shows a block diagram representing an example of a configuration related to a function for space drawing in the drawing device <NUM>. As shown in <FIG>, the drawing device <NUM> includes a processor <NUM>, a memory <NUM>, an I/F (Interface) unit <NUM>, a storage unit <NUM>, a light source <NUM>, and an input device <NUM>. The processor <NUM>, the memory <NUM>, the I/F unit <NUM>, the storage unit <NUM>, the light source <NUM>, and the input device <NUM> are connected to each other via a bus <NUM> such as a system bus or a control bus so that various types of information can be exchanged.

The processor <NUM> reads various programs including a drawing control program <NUM> stored in the storage unit <NUM> into the memory <NUM> and executes processing according to the read program. The memory <NUM> is a work memory for the processor <NUM> to execute the processing. The storage unit <NUM> stores the drawing control program <NUM>, various other information, and the like. Specific examples of the storage unit <NUM> include a hard disk drive (HDD) and a solid state drive (SSD).

The I/F unit <NUM> communicates various types of information with the smartphone <NUM> by wireless communication or wired communication. The input device <NUM> including a drawing switch 28A functions as a user interface. The drawing switch 28A is a switch that is operated to input various instructions in a case where the drawing user performs drawing and is provided on a surface of a main body of the drawing device <NUM>.

The light source <NUM> emits light for drawing, and examples thereof include a light emitting diode (LED). As an example, in the present embodiment, a high luminance type LED is used. Further, in order to improve a light detection rate of the light source <NUM> in the smartphone <NUM>, an LED having an emission color of pure green is adopted instead of a color mixture such as yellow green or the like. Further, since it is preferable that light leakage is small, a light source having high directivity is adopted.

<FIG> shows a functional block diagram representing an example of a configuration related to a function of the drawing device <NUM> according to the present embodiment. As shown in <FIG>, the drawing device <NUM> includes a lighting controller <NUM>, a switch state reception unit <NUM>, and a drawing instruction output unit <NUM>. As an example, in the drawing device <NUM> according to the present embodiment, by the processor <NUM> executing the drawing control program <NUM> stored in the storage unit <NUM>, the processor <NUM> functions as the lighting controller <NUM>, the switch state reception unit <NUM>, and the drawing instruction output unit <NUM>.

The lighting controller <NUM> has a function of controlling lighting and extinguishing of the light source <NUM>. As an example, the lighting controller <NUM> of the present embodiment performs control of lighting the light source <NUM> in a case where a lighting instruction, which is output from the smartphone <NUM>, is input. The lighting instruction of the present embodiment is an example of a light emission instruction signal of the present disclosure. Further, the lighting controller <NUM> performs control of extinguishing the light source <NUM> in a case where an extinguishing instruction, which is output from the smartphone <NUM>, is input.

The switch state reception unit <NUM> has a function of receiving an operation state of the drawing switch 28A operated by the drawing user. As an example, in the present embodiment, in a case where the drawing user starts drawing by using the drawing device <NUM>, the drawing user lights the light source <NUM> and then operates the drawing switch 28A to instruct the drawing device to start drawing. In the following, the fact that the drawing switch 28A is operated to instruct the drawing device <NUM> to start drawing is referred to as "turning on the drawing switch 28A" or the like. Further, in the present embodiment, in a case where the drawing user ends drawing by using the drawing device <NUM>, the drawing user operates the drawing switch 28A to instruct the drawing device <NUM> to end drawing. In the following, the fact that the drawing switch 28Ais operated to instruct the drawing device <NUM> to end drawing is referred to as "turning off the drawing switch 28A" or the like.

The drawing instruction output unit <NUM> has a function of outputting a drawing start instruction and a drawing end instruction according to a state of the drawing switch 28A received by the switch state reception unit <NUM>, to the smartphone <NUM>. Specifically, in a case where the switch state reception unit <NUM> receives that the drawing switch 28A is turned on as the operation state of the drawing switch 28A, the drawing instruction output unit <NUM> outputs the drawing start instruction to the smartphone <NUM>. Further, in a case where the switch state reception unit <NUM> receives that the drawing switch 28A is turned off as the operation state of the drawing switch 28A, the drawing instruction output unit <NUM> outputs the drawing end instruction to the smartphone <NUM>. In the present embodiment, in a case where collectively referring to the drawing start instruction and the drawing end instruction, it is referred to as a "drawing instruction".

Next, the details of the smartphone <NUM> will be described. <FIG> is a block diagram representing an example of a configuration related to a function for the generation of the drawing image in the smartphone <NUM>. As shown in <FIG>, the smartphone <NUM> includes a processor <NUM>, a memory <NUM>, an I/F unit <NUM>, a storage unit <NUM>, a display <NUM>, a camera <NUM>, and an input device <NUM>. The processor <NUM>, the memory <NUM>, the I/F unit <NUM>, the storage unit <NUM>, the display <NUM>, the camera <NUM>, and the input device <NUM> are connected to each other via a bus <NUM> such as a system bus or a control bus so that various types of information can be exchanged.

The processor <NUM> reads various programs including an image generation program <NUM> stored in the storage unit <NUM> into the memory <NUM> and executes processing according to the read program. The memory <NUM> is a work memory for the processor <NUM> to execute the processing. The storage unit <NUM> stores the image generation program <NUM>, various other information, and the like. Specific examples of the storage unit <NUM> include an HDD and an SSD.

The I/F unit <NUM> communicates various types of information with the drawing device <NUM> by wireless communication or wired communication. The display <NUM> and the input device <NUM> function as user interfaces. The display <NUM> displays a captured image obtained being captured by the camera <NUM> in a live view and provides various types of information related to drawing. The display <NUM> is not particularly limited, and examples thereof include a liquid crystal monitor and an LED monitor. Further, the input device <NUM> is operated by the imaging user for inputting various instructions related to drawing. The input device <NUM> is not particularly limited, and examples thereof include a keyboard, a touch pen, and a mouse. The input device <NUM> of the present embodiment includes a shutter button of the camera <NUM>. The smartphone <NUM> adopts a touch panel display in which the display <NUM> and the input device <NUM> are integrated.

<FIG> shows a functional block diagram representing an example of a configuration related to the function of the smartphone <NUM> of the present embodiment. As shown in <FIG>, the smartphone <NUM> includes a drawing mode management unit <NUM>, a drawing device controller <NUM>, an imaging controller <NUM>, a light detection unit <NUM>, a drawing instruction acquisition unit <NUM>, a drawing image generation unit <NUM>, a display controller <NUM>, and a drawing image recording unit <NUM>. As an example, in the smartphone <NUM> according to the present embodiment, by the processor <NUM> executing the image generation program <NUM> stored in the storage unit <NUM>, the processor <NUM> functions as the drawing mode management unit <NUM>, the drawing device controller <NUM>, the imaging controller <NUM>, the light detection unit <NUM>, the drawing instruction acquisition unit <NUM>, the drawing image generation unit <NUM>, the display controller <NUM>, and the drawing image recording unit <NUM>.

The drawing mode management unit <NUM> has a function of managing the transition to the drawing mode and the end of the drawing mode. As an example, in a case where the drawing mode management unit <NUM> of the present embodiment receives a transition instruction to the drawing mode described in detail later, which is performed by the imaging user using the input device <NUM>, the drawing mode management unit <NUM> performs the transition to the drawing mode. Further, in a case where the drawing mode management unit <NUM> receives an end instruction of the drawing mode described in detail later, which is performed by the imaging user using the input device <NUM>, the drawing mode management unit <NUM> ends the drawing mode.

The drawing device controller <NUM> has a function of controlling the drawing device <NUM>. As an example, in a case where the drawing device controller <NUM> of the present embodiment performs the transition to the drawing mode, the drawing device controller <NUM> outputs a lighting instruction for lighting a light source <NUM> of the drawing device <NUM>. Further, in a case where the drawing device controller <NUM> ends the drawing mode, the drawing device controller <NUM> outputs an extinguishing instruction for extinguishing the light source <NUM> of the drawing device <NUM>.

The imaging controller <NUM> includes an exposure controller <NUM> and has a function of controlling a capturing of the captured image by the camera <NUM>. As an example, the camera <NUM> of the present embodiment includes an imaging element (not shown) such as a complementary metal oxide semiconductor (CMOS), and is capable of capturing an RGB color image. In the present embodiment, an image that is captured by the imaging element of the camera <NUM> is referred to as a "captured image". During the drawing mode, the imaging controller <NUM> captures a captured image of a plurality of frames as a moving image.

The exposure controller <NUM> has a function of controlling the exposure in capturing the captured image by the camera <NUM>. As an example, the exposure controller <NUM> of the present embodiment has a function of adjusting the exposure by adjusting at least one of a shutter speed or a stop of the camera <NUM>.

The light detection unit <NUM> has a function of detecting the light of the light source <NUM> of the drawing device <NUM> from the captured image, which is captured by the camera <NUM>. Strictly speaking, the light detection unit <NUM> has a function of detecting an image representing the light of the light source <NUM>, which is included in the captured image. For example, since the drawing user draws in space by moving the drawing device <NUM> in a state where the light source <NUM> is lit, the light detection unit <NUM> detects the trajectory of the light source <NUM> from a series of captured images. In the present embodiment, as described above, the color and the luminance of the light are predetermined for the light source <NUM> of the drawing device <NUM>. Therefore, the light detection unit <NUM> detects the light of the light source <NUM> from the captured image based on the predetermined color and luminance of the light. A detection status of the light detected by the light detection unit <NUM> is output to the light detection unit <NUM> and the drawing image generation unit <NUM>.

The drawing instruction acquisition unit <NUM> has a function of acquiring a drawing instruction output from the drawing device <NUM>. Specifically, the drawing instruction acquisition unit <NUM> acquires a drawing start instruction and a drawing end instruction output from the drawing device <NUM> and input to the smartphone <NUM>. The drawing start instruction and the drawing end instruction, which are acquired by the drawing instruction acquisition unit <NUM>, are output to the drawing image generation unit <NUM>.

The drawing image generation unit <NUM> has a function of generating a drawing image based on the detection status of the light detection unit <NUM>. As an example, the drawing image generation unit <NUM> of the present embodiment generates the drawing image by drawing an image, which follows a drawing condition instructed by the imaging user using the input device <NUM>, according to the trajectory of the light of the light source <NUM> of the drawing device <NUM> detected by the light detection unit <NUM>. The drawing condition, which is instructed by the imaging user using the input device <NUM>, includes at least one of color, lightness, thickness, or a line type such as a dotted line or a solid line of drawn lines, but it is not particularly limited. For example, in a case where "pink", "thick line", "high lightness", and "solid line" is instructed as the drawing condition by the imaging user, the drawing image generation unit <NUM> draws a thick and solid line in high-lightness pink according to the trajectory of the light detected by the light detection unit <NUM>. In the following, for convenience of explanation, a case where the drawing image generation unit <NUM> draws a line drawing according to the trajectory of the light will be described below.

The display controller <NUM> has a function of displaying the drawing image, which is generated by the drawing image generation unit <NUM>, on the display <NUM>. During the drawing mode, the display controller <NUM> of the present embodiment displays the drawing image, which is generated by the drawing image generation unit <NUM>, on the display <NUM> as a live view image of the camera <NUM>.

The drawing image recording unit <NUM> has a function, regarding the drawing image generated by the drawing image generation unit <NUM>, of correcting the brightness of the background region, which is a background of the light detected by the light detection unit <NUM>, and recording the drawing image in which the brightness is corrected to the storage unit <NUM>. As an example, in a case where the drawing mode ends, the drawing image recording unit <NUM> of the present embodiment corrects the brightness of the background region, which became darker due to the reduction of the exposure by the exposure controller <NUM>, by increasing the sensitivity of the drawing image. A method of increasing the sensitivity by the drawing image generation unit <NUM> includes increasing a gain of the drawing image by a predetermined amount. In other words, a method of increasing the sensitivity by the drawing image generation unit <NUM> includes increasing a signal value of each pixel of the drawing image by a predetermined amount. The drawing image recording unit <NUM> may correct the brightness of only the background region of the drawing image or may correct the entire drawing image.

Next, an operation of the space drawing system <NUM> according to the present embodiment will be described.

As shown in <FIG>, in a case where the drawing user performs a drawing using the drawing device <NUM> in a state where the light source <NUM> is lit, the captured image <NUM>, which is captured by the camera <NUM> of the smartphone <NUM>, includes the light of the light source 26A. The captured image <NUM> shown in <FIG> includes a drawing user U1 representing an image of the drawing user, a light source 26A representing an image of the light source <NUM>, a trajectory <NUM> of light representing an image of the trajectory of the light, and the background region <NUM>. In a case where the surrounding environment, which is a background, is relatively bright, and a difference between the brightness of the surroundings and the brightness of the light source <NUM> is relatively small, the light detection unit <NUM> may not be capable of detecting the trajectory <NUM> of the light from the captured image <NUM> in some cases.

Therefore, in a case where the exposure controller <NUM> performs adjustment of reducing the exposure of the entire imaging surface, the captured image <NUM> becomes dark as a whole, as shown in <FIG>. Since the light source <NUM> is brighter and has higher luminance as compared with the surrounding environment, in the captured image <NUM>, the trajectory <NUM> of the light has relatively higher luminance than that of the background region <NUM>. As a result, the light detection unit <NUM> can detect the trajectory <NUM> of the light from the captured image <NUM>.

However, in the captured image <NUM>, the background region <NUM> becomes darker than the original brightness. For example, in the example shown in <FIG>, the background region <NUM> in the captured image <NUM> is darker than the background region <NUM> in the captured image <NUM>. Therefore, in the drawing image <NUM>, which is generated by the drawing image generation unit <NUM> by detecting the trajectory <NUM> of the light from the captured image <NUM>, although a line drawing <NUM> is accurately drawn according to the trajectory <NUM> of the light, the background region <NUM> becomes dark. Therefore, the drawing image recording unit <NUM> corrects and brightens the brightness of the background region <NUM> in the drawing image <NUM> as described above. For example, the drawing image recording unit <NUM> corrects the brightness of the background region <NUM> in the drawing image <NUM> to a brightness that is equal to the brightness of the background region <NUM> in the captured image <NUM> before reducing the exposure, and generates a drawing image <NUM> in which the brightness of the background region <NUM> is corrected. It should be noted that, here, "equal" means that an error or the like is allowed, and is not limited to a perfect match. As shown in <FIG>, in the drawing image <NUM>, the line drawing <NUM> is drawn with high accuracy according to the trajectory <NUM> of the light, and the background region <NUM> becomes bright. The drawing image recording unit <NUM> records the drawing image <NUM> after correcting the brightness of the background region <NUM> in the storage unit <NUM>.

Furthermore, each operation of the smartphone <NUM> and the drawing device <NUM> will be described. <FIG> is a flowchart illustrating an example of an image generation process executed by the smartphone <NUM> and a drawing control process executed by the drawing device <NUM>. The smartphone <NUM> executes the image generation process shown in <FIG> by executing the image generation program <NUM> stored in the storage unit <NUM>. Further, the drawing device <NUM> executes the drawing control process shown in <FIG> by executing the drawing control program <NUM> stored in the storage unit <NUM>.

In a case where drawing is performed by the space drawing system <NUM>, first, the imaging user instructs the input device <NUM> of the smartphone <NUM> to make a transition to the drawing mode.

Therefore, in the smartphone <NUM>, in step S100, the drawing mode management unit <NUM> determines whether or not to make a transition to the drawing mode. The determination in step S100 is determined to be negative until the instruction to make a transition to the drawing mode is performed. On the other hand, in a case where the instruction to make a transition to the drawing mode is performed, the transition to the drawing mode is made, the determination in step S100 is determined to be positive, and the process proceeds to step S102. A period from the next step S102 until the drawing start instruction is input (see step S112) is a drawing standby period.

In step S102, the drawing device controller <NUM> outputs a lighting instruction to the drawing device <NUM>. In a case where the lighting instruction is output from the smartphone <NUM>, in the drawing device <NUM>, the lighting instruction is input in step S200, and the lighting controller <NUM> lights the light source <NUM> in the next step S202.

In the smartphone <NUM>, in the next step S104, the exposure controller <NUM> reduces the exposure of the camera <NUM> as described above. In the present embodiment, the camera <NUM> normally reduces the exposure by a predetermined amount from the standard exposure (standard exposure) specified for imaging according to the imaging environment. Specifically, how much the exposure is reduced may be determined by design or experimentally, depending on the imaging element of the camera <NUM> or the like, and based on the brightness of the entire imaging surface at which the light of the light source <NUM> can be detected.

In the next step S106, the imaging controller <NUM> starts the capture of the captured image by using the camera <NUM>. The camera <NUM> captures a plurality of captured images at a predetermined frame rate in accordance with the control of the imaging controller <NUM>. Further, the imaging controller <NUM> starts the display of the captured image on the display <NUM> as a live view image.

In the next step S108, the light detection unit <NUM> starts the detection of the light of the light source <NUM> from the captured image. In the next step S110, the exposure controller <NUM> performs control to further reduce the exposure of the camera <NUM> in a case where the light detection unit <NUM> cannot detect the light of the light source <NUM> from the captured image. For example, in a case where the imaging environment is too bright compared to the assumed environment, the light of the light source <NUM> may not be detected from the captured image, which is captured by reducing the exposure in step S104. Therefore, in the present embodiment, in step S110, the exposure controller <NUM> performs control to further reduce the exposure of the camera <NUM>. The extent to which the exposure of the camera <NUM> is reduced is not particularly limited, and for example, the exposure may be reduced until the light of the light source <NUM> can be detected from the captured image. Further, for example, in a case where a lower limit value is set and the light of the light source <NUM> cannot be detected from the captured image even after the lower limit value of the exposure reaches, a warning or the like may be notified.

As described above, in the smartphone <NUM>, the exposure of the camera <NUM> is controlled during the drawing standby period, and the light of the light source <NUM> can be detected from the captured image.

On the other hand, in the drawing device <NUM>, in a case where the drawing is ready after the light source <NUM> is lit, the drawing switch 28Ais operated to enter an ON state. In step S204, in a case where the switch state reception unit <NUM> receives the fact that the drawing switch 28A enters an ON state, in the next step S206, the drawing instruction output unit <NUM> outputs the drawing start instruction to the smartphone <NUM>.

In a case where the drawing start instruction is output from the drawing device <NUM>, in the smartphone <NUM>, in step S112, the drawing instruction acquisition unit <NUM> acquires the drawing start instruction input to the smartphone <NUM>. In a case where the drawing instruction acquisition unit <NUM> acquires the drawing start instruction, the drawing standby period ends, the transition is made to the drawing period, and drawing is in progress.

In the next step S114, as described above, the drawing image generation unit <NUM> starts the generation of the drawing image based on the detection status of the light detection unit <NUM>. In the next step S116, the display controller <NUM> starts the display of the drawing image generated by the drawing image generation unit <NUM> on the display <NUM> as a live view image. The drawing image, which is generated in step S114, is an image in which the background region <NUM> is in a dark state as in the drawing image <NUM> (see <FIG>) described above due to the reduction of the exposure of the camera <NUM> in step S108 or the like. Therefore, the background region of the drawing image displayed on the display <NUM> as a live view image is in a dark state.

As described above, in the smartphone <NUM>, during the drawing period, the drawing image generation unit <NUM> generates a drawing image based on the detection status of the light detected by the light detection unit <NUM> from the captured image, which is captured by the camera <NUM>, and the drawing image is displayed on the display <NUM> as a live view image.

On the other hand, in a case where the drawing is ended, or in a case where the drawing is temporarily stopped, the drawing user, who performs a drawing using the drawing device <NUM>, operates the drawing switch 28A to make an OFF state. In step S208, in a case where the switch state reception unit <NUM> receives the fact that the drawing switch 28A enters an OFF state, in the next step S210, the drawing instruction output unit <NUM> outputs the drawing end instruction to the smartphone <NUM>.

In a case where the drawing end instruction is output from the drawing device <NUM>, in the smartphone <NUM>, in step S118, the drawing instruction acquisition unit <NUM> acquires the drawing end instruction input to the smartphone <NUM>. In a case where the drawing instruction acquisition unit <NUM> acquires the drawing end instruction, in the next step S120, the drawing image generation unit <NUM> ends the generation of the drawing image. In a case where the drawing image generation unit <NUM> ends the generation of the drawing image, the drawing period is ended.

In a case where the drawing user, who performs a drawing using the drawing device <NUM>, wants to perform a drawing in another region of the imaging space, or the like, the drawing user temporarily makes the drawing switch 28A an OFF state, moves the drawing device <NUM> to another region where the drawing user wants to draw, makes the drawing switch 28A an ON state again, and starts performing a drawing. Therefore, the switch state reception unit <NUM> of the drawing device <NUM> determines whether or not the fact that the drawing switch 28A is in an ON state is received. In a case where the fact that the drawing switch 28A is in an ON state is not received, the determination in step S212 is determined to be negative, and the process proceeds to step S214. On the other hand, in a case where the fact that the drawing switch 28A is in an ON state is received, the determination in step S212 is determined to be positive, the process returns to step S206, and the processes of steps S206 to S210 are repeated.

Therefore, the smartphone <NUM> determines whether or not the drawing start instruction is input in step S122 after the generation of the drawing image is ended in step S120. As described above, in a case where the drawing user makes the drawing switch 28A in an ON state again, the drawing start instruction is output again from the drawing device <NUM> to the smartphone <NUM>. In a case where the drawing instruction acquisition unit <NUM> acquires the drawing start instruction again, the determination in step S122 is determined to be positive, the process returns to step S114, the transition is made to the drawing period again, and the processes of steps S114 to S120 are repeated. On the other hand, in a case where the drawing instruction acquisition unit <NUM> does not acquire the drawing start instruction, the determination in step S122 is determined to be negative, and the process proceeds to step S124.

In step S124, the drawing mode management unit <NUM> determines whether or not to end the drawing mode. As an example, in the present embodiment, in a case where the imaging user presses a shutter button included in the input device <NUM>, the drawing mode is ended. Therefore, in a case where the shutter button is not pressed, the determination in step S124 is determined to be negative, and the process returns to step S122. On the other hand, in a case where the shutter button is pressed, the determination in step S124 is determined to be positive, and the process proceeds to step S126.

In step S126, the imaging controller <NUM> ends the capturing of the captured image by the camera <NUM>, and the light detection unit <NUM> ends the detection of the light. Further, the display controller <NUM> ends the display of the live view image displayed on the display <NUM>.

In next step S128, the drawing device controller <NUM> outputs an extinguishing instruction to the drawing device <NUM>. In a case where the extinguishing instruction is output from the smartphone <NUM>, in the drawing device <NUM>, the extinguishing instruction is input in step S214, and the lighting controller <NUM> extinguishes the light source <NUM> in the next step S216. In a case where the process in step S216 is ended, the drawing control process shown in <FIG> is ended in the drawing device <NUM>.

On the other hand, in the smartphone <NUM>, in the next step S130, the drawing image recording unit <NUM> corrects the brightness of the background region of the drawing image and records the drawing image, in which the brightness is corrected, in the storage unit <NUM>. As in the drawing image <NUM> (see <FIG>) described above, the drawing image recording unit <NUM> of the present embodiment corrects the brightness of the background region, which became darker due to the reduction of the exposure, to be the same as the brightness of the captured image captured with the standard exposure by increasing the sensitivity of the drawing image. In a case where the process of step S130 is ended, in the smartphone <NUM>, the image generation process shown in <FIG> is ended. The drawing image, in which the brightness is corrected, may be displayed on the display <NUM> before the image generation process is ended.

As described above, the smartphone <NUM> of the present embodiment includes the camera <NUM> that captures the light emitted from the drawing device <NUM> and a processor <NUM> that is configured to generate a drawing image based on the detection status in which the light is detected from the captured image, which is captured by the camera <NUM>. The processor <NUM> is configured to cause the camera <NUM> to change an exposure of the entire imaging surface to capture the captured image, detect the light from the captured image, which is captured in a state in which the exposure is changed, and record the drawing image, which is generated based on the detection status of the light, by correcting a brightness of a background region, which is a background of the light.

As described above, in the smartphone <NUM> of the present embodiment, since the brightness of the background image of the drawing image, which became darker due to the reducing of the exposure, is corrected, it is possible to obtain a high quality drawing image even in the background region.

In the first embodiment, the brightness of the background region of the drawing image is corrected, whereas in the present embodiment, a form of obtaining a drawing image with high quality even in the background region by capturing an image of the background region without reducing the exposure will be described.

Since a configuration of the drawing device <NUM> according to the present embodiment is the same as that of the first embodiment (see <FIG> and <FIG>), the description thereof will be omitted.

On the other hand, the smartphone <NUM> has the same hardware configuration as that of the first embodiment (see <FIG>), but the functional configuration is partially different. <FIG> shows a functional block diagram representing an example of a configuration related to the function of the smartphone <NUM> of the present embodiment. As shown in <FIG>, the smartphone <NUM> differs from the smartphone <NUM> (see <FIG>) of the first embodiment in that a drawing image generation unit <NUM> is provided instead of the drawing image generation unit <NUM> and a composite image generation unit <NUM> is provided instead of the drawing image recording unit <NUM>.

The drawing image generation unit <NUM> of the present embodiment is the same as the drawing image generation unit <NUM> of the first embodiment in that it has a function of generating a drawing image based on a detection status of the light detection unit <NUM>. However, the drawing image generation unit <NUM> of the present embodiment has a function of storing only a drawing corresponding to the detection status of the light as the drawing image. As an example, during the drawing period, the display controller <NUM> of the present embodiment superimposes a drawing image, which is obtained only with a drawing corresponding to the detection status of the light, on the captured image and displays the superimposed image on the display <NUM> as a live view image.

Further, the composite image generation unit <NUM> has a function of generating a composite image obtained by composing a drawing image, which is generated by the drawing image generation unit <NUM> and which includes only a drawing corresponding to the detection status of the light, and a captured image, which is captured by the camera <NUM> with proper exposure, as a drawing image including a background.

An operation of the space drawing system <NUM> according to the present embodiment will be described with reference to <FIG>.

Similar to the description with reference to <FIG> in the first embodiment, in a case where the trajectory <NUM> of the light cannot be detected from the captured image <NUM>, by reducing the exposure of the entire imaging surface and obtaining the captured image <NUM>, the trajectory <NUM> of the light can be detected.

The drawing image generation unit <NUM> of the present embodiment stores only the drawing corresponding to the detection status of the trajectory <NUM> of the light as the drawing image <NUM> as one layer. The drawing image <NUM> does not include a background image.

On the other hand, in a case where the drawing mode is ended, the exposure controller <NUM> restores the exposure to the original state, and then the imaging controller <NUM> causes the camera <NUM> to capture the captured image. Therefore, although it is difficult to see the trajectory <NUM> of the light in the captured image <NUM>, the background region <NUM> has the same brightness as before the exposure of the entire imaging surface is reduced. Therefore, the composite image generation unit <NUM> generates a composite image <NUM> by composing the drawing image <NUM> and the captured image <NUM>. In the composite image <NUM>, the line drawing <NUM> is accurately drawn according to the trajectory <NUM> of the light, and the background region <NUM> has the original brightness.

Furthermore, each operation of the smartphone <NUM> and the drawing device <NUM> of the present embodiment will be described. <FIG> is a flowchart illustrating an example of an image generation process executed by the smartphone <NUM> and a drawing control process executed by the drawing device <NUM>. Note that the drawing control process, which is executed by the drawing device <NUM>, is the same as the drawing control process (see <FIG>) executed by the drawing device <NUM> of the first embodiment, and thus the description thereof will be omitted. On the other hand, an image generation process, which is executed by the smartphone <NUM>, differs from the image generation process executed by the smartphone <NUM> of the first embodiment (see <FIG>) in that step S115 is provided instead of step S114 as shown in <FIG>, a process of step S125 is provided between steps S124 and S126, and a process of step S131 is provided instead of step S130.

In the smartphone <NUM> of the present embodiment, in step S115, in a case where the drawing image generation unit <NUM> starts the generation of the drawing image, as described above, only the drawing corresponding to the detection status of the light is stored as the drawing image in a layer separate from the background image.

Further, in step S125, in a case where the drawing mode is ended, the imaging controller <NUM> restores the exposure of the camera <NUM> by the exposure controller <NUM> to capture the background image as the captured image. As an example, the exposure controller <NUM> restores the exposure of the camera <NUM> to the standard exposure before reducing the exposure in step S104.

Further, in step S131, as mentioned above, the composite image generation unit <NUM> generates a composite image by composing the drawing image, which is generated by the drawing image generation unit <NUM>, which is stored as a separate layer, and which includes only a drawing corresponding to the detection status of the light, and the captured image that is a background image captured in step S125. The generated composite image is stored in the storage unit <NUM>. In a case where the process of step S131 is ended, in the smartphone <NUM>, the image generation process shown in <FIG> is ended.

As described above, the smartphone <NUM> of the present embodiment includes the camera <NUM> that captures the light emitted from the drawing device <NUM> and a processor <NUM> that is configured to generate a drawing image based on the detection status in which the light is detected from the captured image, which is captured by the camera <NUM>. The processor <NUM> is configured to cause the camera <NUM> to reduce the exposure of the entire imaging surface to capture the captured image, cause the camera to capture the background image, which is a background of the light in the captured image, by increasing the exposure, and generate a composite image by superimposing the drawing image drawn based on the detection status in which the light is detected and the background image.

As described above, in the smartphone <NUM> of the present embodiment, since the drawing image of only the drawing corresponding to the detection status of the light and the background image captured with the increased exposure are composed, a high quality drawing image can be obtained even in the background region.

In the first and second embodiments, a form in which the light emitted from the drawing device <NUM> is easily detected by reducing the exposure of the entire imaging surface of the camera <NUM> has been described. In contrast, in the present embodiment, a case where the light emitted from the drawing device <NUM> is easily detected by further increasing the exposure of the entire imaging surface will be described.

On the other hand, the smartphone <NUM> has the same hardware configuration as that of the first embodiment (see <FIG>), but the functions of the exposure controller <NUM> and the drawing image recording unit <NUM> are partially different.

As described above, the exposure controller <NUM> performs adjustment to reduce the exposure of the entire imaging surface in a case where the light of the light source <NUM> cannot be detected or is difficult to be detected from the captured image because the surrounding environment is bright. Further, the exposure controller <NUM> of the present embodiment performs adjustment to increase the exposure of the entire imaging surface in a case where the light of the light source <NUM> cannot be detected or is difficult to be detected from the captured image based on the status of the light source <NUM> of the drawing device <NUM>.

Further, as described above, in a case where the exposure controller <NUM> performs the adjustment of reducing the exposure of the entire imaging surface, and in a case where the drawing mode is ended, the drawing image recording unit <NUM> corrects the brightness of the background region, which became darker due to the reduction of the exposure by the exposure controller <NUM>, by increasing the sensitivity of the drawing image. In a case where the exposure controller <NUM> performs the adjustment of increasing the exposure of the entire imaging surface, and in a case where the drawing mode is ended, the drawing image recording unit <NUM> of the present embodiment corrects the brightness of the background region, which became brighter due to the increase of the exposure by the exposure controller <NUM>, by reducing the sensitivity of the drawing image. A method of reducing the sensitivity by the drawing image generation unit <NUM> includes reducing a gain of the drawing image by a predetermined amount. In other words, a method of reducing the sensitivity by the drawing image generation unit <NUM> includes reducing a signal value of each pixel of the drawing image by a predetermined amount.

As shown in <FIG>, in a case where the drawing user performs a drawing using the drawing device <NUM> in a state where the light source <NUM> is lit, the captured image <NUM>, which is captured by the camera <NUM> of the smartphone <NUM>, includes the light of the light source 26A. The captured image <NUM> shown in <FIG> includes a drawing user U1 representing an image of the drawing user, a light source 26A representing an image of the light source <NUM>, a trajectory <NUM> of light representing an image of the trajectory of the light, and the background region <NUM>. The amount of light emitted from the light source <NUM> to the camera <NUM> may be less than the original amount of light. For example, in a case where the amount of light emitted from the light source <NUM> is less than the original amount due to deterioration over time, failure, or the like, the amount of light emitted from the light source <NUM> to the camera <NUM> becomes less than the original amount of light. Further, even in a case where the amount of emitted light is the same as the original amount of light but the surface of the light source <NUM> is dirty, the amount of light emitted from the light source <NUM> to the camera <NUM> becomes less than the original amount of light. In such a case, the light detection unit <NUM> may not be capable of detecting the trajectory <NUM> of the light from the captured image <NUM>.

Therefore, in a case where the exposure controller <NUM> performs adjustment of increasing the exposure of the entire imaging surface, the captured image <NUM> becomes bright as a whole, as shown in <FIG>. Since the light source <NUM> is brighter and has higher luminance as compared with the surrounding environment, in the background region <NUM>, the trajectory <NUM> of the light has relatively higher luminance than that of the background region <NUM>. As a result, the light detection unit <NUM> can detect the trajectory <NUM> of the light from the captured image <NUM>.

However, in the captured image <NUM>, the background region <NUM> becomes brighter than the original brightness. For example, in the example shown in <FIG>, the background region <NUM> in the captured image <NUM> is brighter than the background region <NUM> in the captured image <NUM>. Therefore, in the drawing image <NUM>, which is generated by the drawing image generation unit <NUM> by detecting the trajectory <NUM> of the light from the captured image <NUM>, although a line drawing <NUM> is accurately drawn according to the trajectory <NUM> of the light, the background region <NUM> becomes dark. Therefore, the drawing image recording unit <NUM> corrects and darkens the brightness of the background region <NUM> in the drawing image <NUM> as described above. For example, the drawing image recording unit <NUM> corrects the brightness of the background region <NUM> in the drawing image <NUM> to a brightness that is equal to the brightness of the background region <NUM> in the captured image <NUM> before increasing the exposure, and generates a drawing image <NUM> in which the brightness of the background region <NUM> is corrected. It should be noted that, here again, "equal" means that an error or the like is allowed, and is not limited to a perfect match. As shown in <FIG>, in the drawing image <NUM>, the line drawing <NUM> is drawn with high accuracy according to the trajectory <NUM> of the light, and the background region <NUM> becomes dark. The drawing image recording unit <NUM> records the drawing image <NUM> after correcting the brightness of the background region <NUM> in the storage unit <NUM>.

Furthermore, each operation of the smartphone <NUM> and the drawing device <NUM> of the present embodiment will be described. <FIG> is a flowchart illustrating an example of an image generation process executed by the smartphone <NUM> and a drawing control process executed by the drawing device <NUM>. Note that the drawing control process, which is executed by the drawing device <NUM>, is the same as the drawing control process (see <FIG>) executed by the drawing device <NUM> of the first embodiment, and thus the description thereof will be omitted. On the other hand, the image generation process, which is executed by the smartphone <NUM>, differs from the image generation process (see <FIG>) executed by the smartphone <NUM> of the first embodiment in that steps S109 and S111 are provided instead of step S110 as shown in <FIG>.

In the smartphone <NUM> of the present embodiment, in a case where the exposure controller <NUM> reduces the exposure of the camera <NUM> as described above in step S104, the exposure is reduced to an assumed lower limit value.

After that, as shown in <FIG>, in step S109, the light detection unit <NUM> determines whether or not the light of the light source <NUM> can be detected from the captured image. In a case where the light of the light source <NUM> can be detected from the captured image, the determination in step S109 is determined to be positive, and the process proceeds to step S112. On the other hand, in a case where the light of the light source <NUM> cannot be detected from the captured image, the determination in step S109 is determined to be negative, and the process proceeds to step S111.

As described above, in a case where the light of the light source <NUM> cannot be detected from the captured image because the amount of light emitted from the light source <NUM> to the camera <NUM> is less than the original amount of light, even in a case where the exposure is reduced in this way, the light of the light source <NUM> cannot be detected from the captured image.

Therefore, in step S111, the exposure controller <NUM> increases the exposure of the camera <NUM> as described above. As an example, in the present embodiment, the camera <NUM> normally increases the exposure to an assumed upper limit value from the standard exposure (standard exposure) specified for imaging according to the imaging environment. The method of increasing the exposure is not limited to the present embodiment. For example, a form of increasing the exposure by a predetermined amount may be used, or specifically how much the exposure is increased may be determined by design or experimentally, depending on the imaging element of the camera <NUM> or the like, and based on the brightness at which the light of the light source <NUM> can be detected. Further, a form may be used in which the exposure may be gradually increased, and whether or not the light of the light source <NUM> can be detected from the captured image is determined each time.

As described above, in the smartphone <NUM> of the present embodiment, the control of reducing the exposure of the camera <NUM> (step S104) or the control of increasing the exposure (step S111) is performed during the drawing standby period, thereby the light of the light source <NUM> can be detected from the captured image.

Therefore, in the present embodiment, in a case where the exposure of the entire imaging surface is reduced by the process in step S104, in step S130, the drawing image recording unit <NUM> corrects the brightness of the background region, which became darker due to the reduction of the exposure, to be equivalent to the brightness of the captured image, which is captured with the standard exposure, by increasing the sensitivity of the drawing image. Further, in a case where the exposure of the entire imaging surface is increased by the process in step S111, the drawing image recording unit <NUM> corrects the brightness of the background region, which became brighter due to the increase of the exposure, to be equivalent to the brightness of the captured image, which is captured with the standard exposure, by reducing the sensitivity of the drawing image. In this way, the drawing image recording unit <NUM> records the drawing image in which the brightness is corrected in the storage unit <NUM>.

As described above, the processor <NUM> of the smartphone <NUM> of the present embodiment can detect the light of the light source <NUM> from the captured image by performing the control of reducing the exposure of the entire imaging surface of the camera <NUM>. Further, in a case where the light of the light source <NUM> cannot be detected from the captured image even in a case where control is performed to reduce the exposure of the entire imaging surface of the camera <NUM>, the processor <NUM> can detect the light of the light source <NUM> from the captured image by performing control of increasing the exposure of the entire imaging surface of the camera <NUM>. Further, the processor <NUM> is configured to cause <NUM> the camera to change an exposure of the entire imaging surface to capture the captured image, detect the light from the captured image, which is captured in a state in which the exposure is changed, and record the drawing image, which is generated based on the detection status of the light, by correcting a brightness of a background region, which is a background of the light.

As described above, in the smartphone <NUM> of the present embodiment, since the brightness of the background image of the drawing image, which became darker due to the reduction of the exposure, is corrected or the brightness of the background image of the drawing image, which became brighter due to increasing of the exposure, is corrected, a high quality drawing image can be obtained even in the background region.

In each of the above-described forms, the case where the drawing device <NUM> is in a form of a light pen has been described, but a form of the drawing device is not particularly limited, and any instruction device may be used as long as it is capable of instructing drawing with light. For example, as shown in <FIG>, a drawing device <NUM> having a print function may be used. The drawing device <NUM> shown in <FIG> has a function of receiving a drawing image generated by the smartphone <NUM> and printing the received drawing image on an instant film.

Further, the timing of reducing the exposure of the entire imaging surface is not limited to the above-described timing. For example, only one of step S108 and step S110 of the image generation process (see <FIG> and <FIG>) may be performed. Further, for example, a form may be used in which the exposure is reduced only in a case where the light cannot be detected after the detection of the light is started. Further, for example, a form may be used in which the exposure is reduced only in a case where the light cannot be detected as appropriate during the drawing. Further, for example, a form may be used in which a control of reducing the exposure to a state where the light can be detected by calibration before making a transition to the drawing mode, is performed. Further, similarly, the timing of increasing the exposure of the entire imaging surface is not limited to the above-described timing. In the above-described third embodiment, the control of increasing the exposure is performed in a case where the light cannot be detected even in a case where the exposure is reduced, but a form may be used in which only the control of increasing the exposure is performed. For example, in a case where the imaging environment is dark, a form may be used in which only the control of increasing the exposure is performed. Further, for example, a form may be used in which the exposure is increased only in a case where the light cannot be detected after the detection of the light is started. Further, for example, a form may be used in which the exposure is increased only in a case where the light cannot be detected as appropriate during the drawing. Further, for example, a form may be used in which a control of increasing the exposure to a state where the light can be detected by calibration before making a transition to the drawing mode, is performed.

Further, in each of the above-described forms, the embodiment in which the smartphone <NUM> controls the lighting and extinguishing of the light source <NUM> of the drawing device <NUM> has been described, but the present embodiments are not limited thereto, and the drawing device <NUM> itself may be configured to control at least one of lighting or extinguishing of the light source <NUM>.

Further, in the above-described form, for example, the following various processors can be used as a hardware structure of a processing unit that executes various processes such as the lighting controller <NUM>, the switch state reception unit <NUM>, and the drawing instruction output unit <NUM> of the drawing device <NUM>, or the drawing mode management unit <NUM>, the drawing device controller <NUM>, the imaging controller <NUM>, the exposure controller <NUM>, the light detection unit <NUM>, the drawing instruction acquisition unit <NUM>, the drawing image generation unit <NUM>, the drawing image generation unit <NUM>, the display controller <NUM>, the drawing image recording unit <NUM>, and the composite image generation unit <NUM> of the smartphone <NUM>. As described above, the above described various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration is able to be changed after manufacturing such as a field programmable gate array (FPGA), a dedicated electric circuit, which is a processor having a circuit configuration specially designed to execute specific processing such as an application specific integrated circuit (ASIC), and the like.

One processing unit may be composed of one of these various processors or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). Further, a plurality of processing units may be composed of one processor.

As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Second, as represented by a system on chip (SoC) or the like, there is a form in which a processor, which implements the functions of the entire system including a plurality of processing units with one integrated circuit (IC) chip, is used. In this way, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined can be used.

Claim 1:
An image generation device (<NUM>) comprising:
a camera (<NUM>) that is configured to capture an image including light emitted from an instruction device (<NUM>); and
a processor (<NUM>) that is configured to generate a drawing image based on a detection status in which the light is detected from the captured image,
wherein, to generate the drawing image, the processor (<NUM>) is configured to
cause the camera to reduce an exposure of an entire imaging surface to capture the captured image, by a predetermined amount, from a standard exposure specified for imaging according to an imagining environment of the camera,
detect the light from the captured image, which is captured in a state in which the exposure is reduced by the predetermined amount, and
record the drawing image, which is generated based on the detection status of the light, by correcting a brightness of a background region, which is a background of the detected light.