Patent Description:
Various display devices have recently been developed, such as fixed display devices including TV receiver (hereinafter simply referred to as TV) and projectors, and mobile display devices including smartphones and notebook PCs (Personal Computers). Further, with the development of the display devices, interfaces and operation methods for controlling the devices have been diversified.

For example, with regard to projectors, in addition to projectors that project images onto screens perpendicular to the ground, as represented by home theaters which have currently been used, the rise of projection mapping techniques have increased the number of cases in which images are projected onto any places such as table surfaces, walls, ceilings, and the like.

Further, in some cases, presentations are made using pointers or the like for pointing images projected by projectors. In these cases, pointers that output infrared light may be utilized. Moreover, systems have also been proposed each of which displays (projects), using a digital pen that outputs IR (infrared spectroscopy) light, a trajectory of the digital pen by a projector. In particular, the systems are useful for digitization of blackboard writing in the education field.

For example, the following PTL <NUM> discloses a technique of imaging an optical signal (infrared light) outputted from an instruction pen by an IR camera mounted on a projector, and, upon detecting an optical signal having a signal level above a predetermined threshold value, outputting position information of the optical signal. PTL <NUM> discloses an information display system including a pointing device and a control device that detects a locus of a tip of the pointing device. The pointing device includes a light emitting unit that projects radiation light extending radially having the tip as a center on a predetermined plane. A camera picks up an image of an area including the radiation light on the predetermined plane. A radiation light detection unit detects the radiation light from the picked-up image. An estimation unit estimates a position of the tip of the pointing device from the detected radiation light.

Here, even if an operating tool such as an electronic pen that outputs IR light operates normally, a case may occur where a camera is unable to detect the IR light of the electronic pen, because a user (operator) himself/herself shields the IR light of the electronic pen depending on a positional relationship between a projection area, the user, and the camera. However, the typical user who does not grasp a position of the camera or a mechanism of the detection is unable to understand why drawing has stopped, and thus, inconvenience is imposed on the user.

According to the present disclosure, there is proposed an information processing device including a controller configured to perform a recognition process of recognizing, on a basis of an image captured by an imaging device imaging a projection area, a position of an operating tool inside the projection area, perform an operation detection process of detecting, on a basis of sensor data outputted by a sensor included in the operating tool, that an operation is performed on the operating tool, and perform, in a case where the operation performed on the operating tool is detected by the operation detection process and the position of the operating tool is not recognized by the recognition process, a projection control process of causing a projection section to project an image for guiding the operating tool to a position which is inside the projection area and where the position of the operating tool is recognizable by the recognition process.

According to the present disclosure, there is proposed an information processing method performed by a processor, the method including: recognizing, on a basis of a captured image obtained by imaging a projection area, a position of an operating tool inside the projection area; detecting, on a basis of sensor data outputted by a sensor included in the operating tool, that an operation is performed on the operating tool; and, in a case where the operation performed on the operating tool is detected and the position of the operating tool is not recognized, causing a projection section to project an image for guiding the operating tool to a position which is inside the projection area and where the position of the operating tool is recognizable.

According to the present disclosure, there is proposed a program for causing a computer to function as a controller that performs a recognition process of recognizing, on a basis of a captured image obtained by imaging a projection area, a position of an operating tool inside the projection area, an operation detection process of detecting, on a basis of sensor data outputted by a sensor included in the operating tool, that an operation is performed on the operating tool, and, in a case where the operation performed on the operating tool is detected by the operation detection process and the position of the operating tool is not recognized by the recognition process, a projection control process of causing a projection section to project an image for guiding the operating tool to a position which is inside the projection area and where the position of the operating tool is recognizable by the recognition process.

In the following, some embodiments of the present disclosure are described in detail with reference to the drawings. It is to be noted that, in the following embodiments, components that have substantially the same functional configuration are indicated by the same reference signs, and thus redundant description thereof is omitted.

It is to be noted that description is given in the following order.

<FIG> is a diagram illustrating an outline of an information processing system according to an embodiment of the present disclosure. The information processing system according to the present embodiment includes, as illustrated in <FIG>, a projector <NUM> that projects an image, a camera <NUM> that images a projection area <NUM> (in <FIG>, an area shot by the camera <NUM> (so-called angle of view) is illustrated as a recognition area <NUM>), and an information processing device <NUM> that controls the projection of the image performed by the projector <NUM>.

Further, when a user performs a drawing operation on the projection area <NUM> using an operating tool <NUM> (so-called electronic pen), a light-emitting portion (e.g., an IR_LED (Light Emitting Diode)) provided at a pen tip of the operating tool <NUM> emits light (specifically, the light-emitting portion emits light by a pressing force of the pen tip, for example), and it becomes possible for the information processing device <NUM> to detect a light-emitting part from a captured image taken by the camera <NUM> and recognize a position (i.e., a drawing position) of the operating tool <NUM>. The camera <NUM> is preset as an imaging range (the recognition area <NUM>) that is a range equal to or slightly larger than the projection area <NUM> of the projector <NUM>, and images light (e.g., IR light) emitted by the operating tool <NUM>. The information processing device <NUM> continuously recognizes the position of the operating tool <NUM> on the basis of the captured image taken by the camera <NUM>, generates a trajectory image L indicating a trajectory of the drawing operation with respect to the operating tool <NUM>, and causes the trajectory image L to be projected by the projector <NUM>.

As a result, the present system is able to freely perform drawing using the operating tool <NUM> on any place (the projection area <NUM>) such as a table surface, a wall, a floor, or the like.

It is to be noted that, the light-emitting portion of the operating tool <NUM> may be achieved by the IR_LED. In this case, to avoid a false operation due to light other than IR light emitted by the operating tool <NUM>, the camera <NUM> may be configured to perform imaging via a visible-light-cut filter (i.e., using an IR camera as the camera <NUM>). In such a case, only the IR light of the pen tip is detected from the captured image taken by the camera <NUM>.

However, if the IR light is shielded by any shielding object or the drawing is performed outside the angle of view of the camera <NUM> (outside the recognition area <NUM> illustrated in <FIG>), the system is unable to detect the IR light and to project the trajectory image.

Here, description will be given of an issue of shielding assumed upon performing drawing using the operating tool <NUM>.

<FIG> is a diagram illustrating shielding (shielding of the projector <NUM>) in a case where a user is located between the projector <NUM> and a projection surface (projection area <NUM>) according to the present embodiment. In an example illustrated in <FIG>, a shadow of the user causes a projection-unavailable area (shield area <NUM>) to be generated in the projection area <NUM>. Usually, in a case where the projector <NUM> performs projection, a shadow of the user appears, so that the user is relatively easily aware that he/she is shielding the projector <NUM>.

In contrast, it is difficult for the user to notice that he/she is shielding the camera <NUM> because the shadow does not appear in the case of shielding the camera <NUM>. Hereinafter, a description will be given with reference to <FIG>.

<FIG> is a diagram illustrating the shielding (shielding of the camera <NUM>) in a case where the user is located between the camera and an operating tool according to the present embodiment. In this case, an area (shield area <NUM>) is generated in which the IR light emission of the operating tool <NUM> is undetectable since the IR light emission is hidden by the user's body. In such a case, the information processing device <NUM> is unable to recognize the position (trajectory) of the IR light emission and unable to generate the trajectory image L; thus, the user is unable to perform drawing even if the operating tool <NUM> is normally operated. It is to be noted that no shadow appears in the shield area <NUM> as in the case of shielding the projector <NUM>; therefore, it is difficult for the user to notice that he himself or she herself is a shielding object.

Further, in the case of the system that projects a trajectory image, it is difficult to understand where the image can be drawn in a situation where nothing is drawn, and there may occur a case where the drawing is started outside the angle of view (outside the recognition area) of the camera <NUM> and is unable to be executed.

Regarding such an issue that the operating tool <NUM> which is operating normally is unrecognizable by the camera <NUM> due to shielding, etc., no feedback to users has been conducted, and the user has had to determine by himself/herself why the drawing has become unavailable. In some cases, this may be improved by actions such as checking the position of the camera <NUM> and changing the standing position of the user. However, it is difficult to distinguish whether the drawing is unavailable because the user does not know a mechanism of the detection or the drawing is unavailable due to other reasons (e.g., not shielding caused by the user but running out of battery of the operating tool <NUM>, shielding caused by other furniture or movable objects (curtains, etc.), and the like), and thus, inconveniences have been imposed.

Accordingly, the present disclosure proposes a mechanism that is able to enhance convenience of the user by guiding the user to an appropriately recognizable area in the case where the position of the operating tool that is being operated is in an unrecognizable area.

<FIG> is a block diagram illustrating an example of an overall configuration of the information processing system according to the present embodiment. As illustrated in <FIG>, the information processing system according to the present embodiment includes the information processing device <NUM>, an output device <NUM>, the camera <NUM>, and the operating tool <NUM>.

The output device <NUM> includes the projector <NUM>, a TV <NUM>, a tablet <NUM>, a smartphone <NUM>, and a PC <NUM>. It is to be noted that the present system may include one or more of those devices in combination as the output device <NUM>, or the system may include a plurality of devices of one type out of those devices. In the example illustrated in <FIG>, a system configuration including the projector <NUM> is illustrated as an example.

The projector <NUM> is a projector device that projects an image anywhere in a space. The projector <NUM> may be, for example, a fixed wide-angle projector or may be a so-called moving projector including a movable portion that is able to change a projection direction, such as a Pan/Tilt drive type. The projection surface is not limited to a plane, and may be a curved surface or may be divided into a plurality of surfaces. Further, the projector <NUM> may be achieved by a plurality of units.

The TV <NUM> is a device that outputs images and sounds by receiving radio waves of television broadcasting. The tablet <NUM> is typically a mobile device that has a screen larger than a screen of the smartphone <NUM>, is able to perform radio communication, and is able to output images, sounds, vibrations, and the like. The smartphone <NUM> is typically a mobile device that has a screen smaller than the screen of the tablet <NUM>, is able to perform radio communication, and is able to output images, sounds, vibrations, and the like. The PC <NUM> may be a fixed desktop PC, a mobile laptop PC, and is able to output images, sounds, and the like.

The output device <NUM> outputs information on the basis of control performed by the information processing device <NUM>. In addition to contents of the information to be outputted, the information processing device <NUM> is also able to control an output method. For example, the information processing device <NUM> is able to control the projection direction of the projector <NUM>.

Further, the output device <NUM> may include a component that is able to perform any outputting other than those described above. For example, the output device <NUM> may include a speaker and a unidirectional speaker. The speaker converts audio data into analog signals via DAC (Digital Analog Converter) and an amplifier, and performs outputting (playing back). The unidirectional speaker is a speaker that is able to form directivity in a single direction. The information processing device <NUM> may control the directivity of the unidirectional speaker.

Further, the output device <NUM> may include an illumination device, an air conditioner, a music playback device, or the like.

The camera <NUM> is an IR recognition camera for recognizing IR LED provided to the operating tool <NUM>, and is an exemplary sensor device that is able to perform imaging via an IR filters and detect only IR light.

In addition to the camera <NUM>, the present system may include other sensors that sense various types of information. For example, the present system may include a RGB camera, a depth sensor, a microphone, etc. to sense information regarding the user and a space in which the user is located. The depth sensor is a device that acquires depth information such as an infrared-ray distance measurement device, an ultrasonic ranging device, LiDAR (Laser Imaging Detection and Ranging), or a stereoscopic camera. In addition, a ToF (Time Of Flight) camera that is able to acquire high-precision range images may be used. The microphone is a device that collects ambient sounds and outputs audio data converted into digital signals via an amplifier and ADC (Analog Digital Converter). The microphone may be an array microphone.

The sensor device senses information on the basis of control performed by the information processing device <NUM>. For example, the information processing device <NUM> may control a zooming rate and an imaging direction of the camera <NUM>.

Further, the sensor device may further include a component that is able to perform any sensing other than those described above. For example, the sensor device may include devices to which information is inputted by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, a lever, and the like. Moreover, the sensor device may also include various sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, an optical sensor, an illuminance sensor, a force sensor, an ultrasonic sensor, a barometric sensor, a gas sensor (Co2), a thermographic camera (far-infrared camera), and the like.

The number of output devices described above and the number of sensor devices described above may be multiple.

The operating tool <NUM> is held by the user and is used when the user draws a trajectory on the projection area <NUM>. Specifically, an IR_LED <NUM> is provided at a tip (pen tip) of the operating tool <NUM> as an example of the light-emitting portion, and when drawing is performed by bringing the pen tip in contact with the projection area <NUM> such as a wall, a switch <NUM> (sensor) is turned on by pushing the pen tip, and the IR_LED <NUM> is controlled to be turned on by a CPU <NUM>. In contrast, when the pen tip is separated from the projection area <NUM> such as the wall, the pen tip is released from being pushed, the switch <NUM> is turned off, and the IR_LED <NUM> is controlled to be turned off by the CPU <NUM>.

It is to be noted that although an example has been described in which the IR_LED <NUM> emits light by pressing force the pen tip on the grounding surface, the present embodiment is not limited thereto, and the IR_LED <NUM> may emit light by detecting an operation of a button, a switch, or the like provided to the operating tool <NUM>, or detecting an operation with respect to the operating tool using a motion sensor included in the operating tool.

Further, the operating tool <NUM> includes a communication module <NUM>. The communication module <NUM> may be coupled to the information processing device <NUM> via wire or radio to transmit and receive data. For example, the communication module <NUM> communicates with the information processing device <NUM> by wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark, Bluetooth (registered trademark), near field communication, and the like.

The CPU <NUM> transmits, to the information processing device <NUM>, a predetermined command based on a user's operation on a button or a switch provided to the operating tool <NUM>, movement of the pen, or the like. Specifically, the CPU <NUM> transmits sensor data (i.e., data indicating that the switch is turned on by pushing the tip, indicating that an operation (drawing operation) has been performed on the operating tool <NUM>) detected by the switch <NUM> from the communication module <NUM> to the information processing device <NUM> on a real-time basis.

It is to be noted that, although the case is mainly described in which the drawing operation of drawing the trajectory using the operating tool <NUM> here as an example, the operation using the operating tool <NUM> according to the present embodiment is not limited thereto, and it is also possible to perform various operations such as clicking operations, dragging and dropping operations, turning pages, scrolling operations, and the like on the displayed images.

The information processing device <NUM> includes an I/F (Interface) <NUM>, a light-emission recognizer <NUM>, a storage <NUM>, an operation detector <NUM>, an image generator <NUM>, a coordinate converter <NUM>, a display controller <NUM>, and a user position recognizer <NUM>.

The I/F <NUM> is a coupling device that couples the information processing device <NUM> to other devices. The I/F <NUM> is achieved by, for example, a USB (Universal Serial Bus) connector, a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a portable communication network (LTE (Long Term Evolution), <NUM> (third generation mobile communication system)), and the like, and performs input and output of information to and from the output device <NUM>, the camera <NUM>, and the operating tool <NUM>. By performing communication with these devices via the I/F <NUM>, the information processing device <NUM> is able to control the output device <NUM>, the camera <NUM>, and the operating tool <NUM>. More specifically, for example, the information processing device <NUM> may perform these controls by outputting control signals to the output device <NUM>, the camera <NUM>, and the operating tool <NUM> via the I/F <NUM>.

The light-emission recognizer <NUM> detects IR light from a captured image (IR captured image) imaged by the camera <NUM>, recognizes position coordinates (e.g., position coordinates of a bright spot) of the detected IR light, accumulates the position coordinates in the storage <NUM>, and outputs the fact that the IR light has been detected (e.g., the fact that a position of the bright spot has been recognized) to the image generator <NUM>. Further, the light-emission recognizer <NUM> also has a function (tracking function) of tracking traveling of the detected IR light and acquiring a traj ectory of the IR light.

The storage <NUM> stores the light-emission position coordinates recognized by the light-emission recognizer <NUM>. The storage <NUM> may also store installation position information of the output device <NUM> and the camera <NUM>, space recognition information, user position information, and various sensor data. The space recognition information is information acquired/recognized by the information processing device <NUM> from a projection space (projection environment) on the basis of various sensor data acquired from a sensor device (not illustrated) (a captured image (visible light image, infrared image) taken by a camera or a bird's-eye camera, depth information acquired by a depth sensor, distance information acquired by a distance measurement sensor, temperature information acquired by a thermographic sensor, voice information acquired by a microphone, and the like). Specifically, for example, the information processing device <NUM> may recognize: a three-dimensional shape of the projection space (projection environment); a three-dimensional shape, a three-dimensional position, and a projectable area (such as a plane area having a predetermined size) of a real object present in the projection space; a three-dimensional position, a posture, a gesture, an utterance, and the like of the user. The information processing device <NUM> thereafter may store them in the storage <NUM> as the space recognition information.

The operation detector <NUM> has a function to detect that an operation has been performed on the operating tool on the basis of the sensor data received from the operating tool <NUM>. Specifically, for example, upon receiving sensor data (i.e., data indicating that the switch is turned on by pushing the tip) detected by the switch <NUM> provided to the operating tool <NUM>, the operation detector <NUM> is able to detect that an operation (drawing operation in this case) has been performed on the operating tool <NUM>. Further, the operation detector <NUM> outputs the operation detection result to the image generator <NUM>.

The image generator <NUM> generates an image to be displayed by the output device <NUM>. For example, the image generator <NUM> generates an image (trajectory image L) displaying a trajectory drawn by the operating tool <NUM> in response to the drawing operation performed by the operating tool <NUM>. Specifically, the image generator <NUM> generates, on the basis of display coordinate information corresponding to the trajectory of the IR light acquired from the coordinate converter <NUM>, an image that displays the trajectory and outputs the image to the display controller <NUM>.

Further, the image generator <NUM> according to the present embodiment generates an image (guidance image) for guiding the operating tool <NUM> to an area (hereinafter also referred to as "drawable area") within the projection area <NUM> and recognizable by the camera <NUM>, in a case where the light emission is not recognized (i.e., the position of the operating tool <NUM> is not recognized) by the light-emission recognizer <NUM> even though the operation (drawing operation in this case) on the operating tool <NUM> is detected by the operation detector <NUM>. In the case where the light emission is not recognized even though the operation on the operating tool <NUM> is detected, there is assumed a case where the user performs the operation outside the recognition area <NUM> of the camera <NUM>, or a case where the IR light of the operating tool <NUM> is shielded by the user, furniture, or movable objects (curtains, etc.), for example.

Here, <FIG> illustrates an example of the guidance image according to the present embodiment. As described above, in the case where the light emission is not recognized even though the operation on the operating tool <NUM> is detected, since the operating tool <NUM> is in a normally usable state (not running out of battery or the like) and in a drawable state, it is desirable to display an illustration, an icon, a guide, or the like that indicates, in an intuitively understandable manner, the fact that it is possible to perform drawing, as the guidance image. Specifically, for example, illustration images and icons of pens as illustrated in guidance images 422a and 422b of <FIG>, or a guidance display for prompting drawing such as "Let's draw!" or the like may be displayed on the drawable area.

In response to a request from the image generator <NUM>, the coordinate converter <NUM> calls information of a trajectory of a target from the storage <NUM>, calculates coordinates at which an image for the target is to be displayed from the information of the trajectory, and outputs the calculated coordinates to the image generator <NUM>.

The display controller <NUM> controls outputting of the image generated by the image generator <NUM> to the output device <NUM> via the I/F <NUM> and displaying of the image to be performed by the output device <NUM>. For example, the display controller <NUM> performs control to cause the trajectory image generated by the image generator <NUM> to be projected by the projector <NUM>. Further, the display controller <NUM> performs control to cause the guidance image generated by the image generator <NUM> to be projected by the projector <NUM>. The guidance image may be displayed at a predetermined position such as an edge of the drawable area, may be displayed at a position where the IR light has been finally recognized, or may be displayed at a position corresponding to an installation direction of the camera <NUM> or the projector <NUM> (e.g., at a right edge in a case where the camera <NUM> is located in the right direction of the drawable area). This makes it possible to make a guidance to an appropriately recognizable area, to enhance the convenience of the user.

The user position recognizer <NUM> has a function of recognizing a position of the user on the basis of a captured image. For example, in a case where the present system uses an RGB IR camera as the camera <NUM>, it is possible to acquire a visible light image and an IR image simultaneously. In this case, the user position recognizer <NUM> is able to recognize a position of a person from the visible light image by bone estimation or the like. On the basis of the recognized position of the user and whether or not the light emission recognition based on the IR image is possible, the information processing device <NUM> is able to determine that the light emission of the operating tool <NUM> is shielded by the user. It is to be noted that upon performing the bone estimation based on the visible light image, it is also possible to acquire not only a bone position but also an area of the human body by using an inter-frame difference method. Still further, a thermographic camera, a depth camera, or a ToF sensor may be prepared in addition to the IR camera for recognizing light emission, thereby extracting a contour (area of the human body) of the user (it is desirable that an optical axis of these sensors be as close as possible to the IR camera).

The light-emission recognizer <NUM>, the operation detector <NUM>, the image generator <NUM>, the coordinate converter <NUM>, the display controller <NUM>, and the user position recognizer <NUM> described above may be achieved by a controller (not illustrated) provided to the information processing device <NUM>. The controller functions as an arithmetic processing unit and a control unit, and controls overall operations in the information processing device <NUM> in accordance with various programs. The controller is achieved, for example, by an electronic circuit such as CPU (Central Processing Unit) or a microprocessor. Further, the controller may include a ROM (Read Only Memory) that stores programs and arithmetic parameters to be used, and a RAM (Random Access Memory) that temporarily stores parameters which vary as appropriate.

Further, the configuration of the information processing device <NUM> according to the present embodiment is not limited to the example illustrated in <FIG>.

For example, the information processing device <NUM> may be configured by a plurality of devices, or at least some of the functional configurations of the information processing device <NUM> illustrated in <FIG> may be provided to the output device <NUM>, the camera <NUM>, or the operating tool <NUM>. Moreover, the information processing device <NUM> does not necessarily have all the functional configurations illustrated in <FIG>.

In addition, the information processing device <NUM> may also be achieved by a PC, a smartphone, an edge server, an intermediate server, or a cloud server.

Subsequently, an example of a flow of a display process of the guidance image according to the present embodiment will be described with reference to the drawings.

<FIG> is a flowchart illustrating an example of a flow of a display process of the guidance image according to the present embodiment. As illustrated in <FIG>, first, the information processing device <NUM> determines whether or not the switch <NUM> of the pen tip is pushed (by the drawing operation) by the operation detector <NUM> on the basis of sensor data transmitted from the operating tool <NUM> (step S103).

Next, if it is determined that the switch <NUM> of the pen tip is pushed (step S103/Yes), the light-emission recognizer <NUM> recognizes the IR light from the captured image (IR image) of the camera <NUM> (step S106). Here, as an example, the IR light-recognition process is performed upon detecting that the switch <NUM> of the pen tip is pushed. However, the present embodiment is not limited thereto, and the IR light-recognition process may be performed continuously (at all times).

Thereafter, if the IR light is recognized (step S106/Yes), the information processing device <NUM> performs trajectory drawing (step S109). Specifically, the information processing device <NUM> generates a trajectory image in accordance with the position coordinates of the IR light by the image generator <NUM>, and performs control to cause the projector <NUM> to display (project) the trajectory image by the display controller <NUM>.

In contrast, if the IR light is not recognized (step S106/No), the information processing device <NUM> displays a UI (guidance image) for guiding the operating tool <NUM> to a writable place (drawable area within the projection area <NUM> and recognizable by the camera <NUM>) at the writable place (step S112). Specifically, the information processing device <NUM> generates a guidance image (or acquire a guidance image prepared in advance), and performs control to cause the projector <NUM> to display (project) the guidance image by the display controller <NUM>. As a result, the user is able to intuitively understand that it is not running out of battery of the operating tool <NUM> or a system trouble and it is possible to perform the drawing, and that it is possible to perform the drawing in the periphery where the guidance image is displayed.

Thereafter, the information processing device <NUM> repeats the process illustrated in steps S106 to S112 described above while the switch <NUM> of the pen tip continues to be pushed (while the operation in the operating tool <NUM> continues to be detected by the operation detector <NUM>) (step S115).

If the information processing device <NUM> according to the present embodiment further determines, on the basis of a position of the user, that a factor of not being able to recognize the IR light is shielding caused by the user, the information processing device <NUM> is able to give notice that the shielding caused by the user is taking place as the guidance image and prompt the user to take an appropriate action.

<FIG> is a flowchart illustrating an example of a flow of a display process of a guidance image corresponding to a factor of non-recognizability.

In steps S103 to S <NUM> illustrated in <FIG>, a process similar to the steps described referring to <FIG> is performed. In other words, if the switch <NUM> of the pen tip of the operating tool <NUM> is pushed (by the drawing operation) (step S103), the information processing device <NUM> recognizes the IR light (step S106), and performs trajectory drawing (generation and displaying of the trajectory image) (step S109).

Next, the information processing device <NUM> determines whether or not the IR light has become unrecognizable at a user position (step S120). Specifically, it is appreciated that the IR light is shielded by the user, in a case where the IR light has become unrecognizable by the light-emission recognizer <NUM> and a place where the IR light has become unrecognizable is the user position recognized by the user position recognizer <NUM> even though the drawing operation is detected by the operation detector <NUM>. <FIG> is a diagram illustrating recognition of the user position and the shielding caused by the user. For example, in a case where the present system uses the RGB IR camera as the camera <NUM>, a visible light image <NUM> as illustrated in <FIG> may be acquired, and the user position recognizer <NUM> recognizes the user position by bone estimation or the like on the basis of the visible light image <NUM>. Thereafter, in a case where the IR light tracked by the light-emission recognizer <NUM> moves toward a region recognized as the user position and becomes unrecognizable (lost) (in the middle of drawing), the image generator <NUM> is able to determine that the IR light is shielded by the user.

Next, if the IR light has become unrecognizable at the user position (step S120/Yes), the image generator <NUM> determines that the IR light is shielded by the user, generates an image (guidance image) which gives notice that the IR light is shielded by the user, and the display controller <NUM> controls the generated guidance image to be projected by the projector <NUM> (step S123). Here, <FIG> illustrates an example of the guidance image that gives notice of the shielding caused by the user according to the present embodiment. For example, a guidance image 422d of <FIG> is an icon image indicating how the user enters into an angle of view of the camera and is standing in the way. Further, a guidance image 422e of <FIG> is a guidance display such as "Your shadow is in the way". By displaying such a guidance image, the user is able to intuitively understand that it is not possible to perform drawing (to be recognized) due to the standing position of the user, and is able to make an improvement by confirming the position of the camera <NUM> and changing the standing position of the user.

Further, the display controller <NUM> controls the guidance image to be displayed on the drawable area. Referring now to <FIG>, the drawable area according to the present embodiment will be described. As illustrated in <FIG>, the recognition area <NUM> corresponding to the imaging range of the camera <NUM> may be preset to be a range that is equal to or slightly larger than the projection area <NUM>. In addition, in the present embodiment, an area that is the projection area <NUM> and the recognition area <NUM>, and further excludes the shield area <NUM> (area in which the user shields the projection area <NUM> of the projector <NUM>) and the shield area <NUM> (area in which the user shields the recognition area <NUM> of the camera <NUM>) is defined as a drawable area <NUM>. The display controller <NUM> may display the guidance image <NUM> at a predetermined position such as the edge of the drawable area <NUM>, at a position at which the IR light has been recognizable for the last time, or at a position corresponding to the installation direction of the camera <NUM> or the projector <NUM> (e.g., the right edge when the camera <NUM> is located in the right direction of the drawable area).

It is possible to appropriately calculate the shield area <NUM> and the shield area <NUM> on the basis of a visible light image acquired by a visible-light camera, a three-dimensional position of the user, a three-dimensional position of the camera <NUM>, a three-dimensional position of the projector <NUM>, a three-dimensional position of the projection area <NUM>, a three-dimensional position of the recognition area <NUM>, a positional relationships and distances therebetween, etc..

Here, although a real shadow is generated in the shield area <NUM>, no shadow is generated in the shield area <NUM> where the camera <NUM> is shielded; thus, the information processing device <NUM> may generate a virtual shadow image indicating the shield area <NUM> (at least a contour of the shield area <NUM>) as the guidance image by the image generator <NUM> and display the virtual shadow image. <FIG> is a diagram illustrating a case where the virtual shadow image is displayed as the guidance image. As illustrated in <FIG>, in the drawable area <NUM>, the real shadow appears in the shield area <NUM> where the user shields the projection by the projector <NUM>, while no shadow appears in the shield area <NUM> where the user shields the imaging by the camera <NUM>; thus, a virtual shadow image 422f is generated and the projector <NUM> displays the virtual shadow image 422f at a position of the shield area <NUM>. As a result, the user is able to intuitively understand that it is not possible to perform the drawing in the area of the shadow of the user, and to make an improvement by changing the standing position or the like. It is to be noted that an example has here been described in which the image indicating the contour of the shield area <NUM> is generated and displayed as the guidance image, but the present embodiment is not limited thereto. For example, the image generator <NUM> of the information processing device <NUM> may generate an image (e.g., a simple rectangular image including the shield area <NUM>) corresponding to the position and a size of the shield area <NUM> as the guidance image. For example, in a case where a process that is able to reduce a process load compared to contour extraction is suitable, the image generator <NUM> of the information processing device <NUM> may generate and display a rectangular image corresponding to the position and the size of the shield area.

Further, the shadow of the user normally appears in the case of the projection performed by the projector <NUM>; therefore, it is relatively easy for the user to notice that the user is shielding the projector <NUM>. However, in a case where a background color of an image to be projected is black and information to be displayed is white, a shadow does not appear, because nothing is projected on projection area by default. In this case, the information processing device <NUM> may also display a virtual shadow on the shield area <NUM> of the projector <NUM>. The shield area <NUM> of the projector <NUM> may be calculated on the basis of the position of the user, an installation position the projector <NUM>, and a place of projection area. Alternatively, in a case where the projector <NUM> and the camera <NUM> are installed close to each other and the optical axes of thereof are as close as possible or coaxial with each other, the shield area <NUM> of the camera <NUM> is expressed as a virtual shadow, thereby causing the user to recognize the virtual shadow also as the shadow of the projector <NUM>. It is to be noted that also in this case, the guidance image indicating the shield area <NUM> of the projector <NUM> is not limited to the image of contour extraction. For example, the image generator <NUM> of the information processing device <NUM> may generate an image corresponding to a position and a size of the shield area <NUM> (e.g., a simple rectangular image including the shield area <NUM>).

Thereafter, if the shielding is improved by changing the standing position of the user or the like (i.e., in a case where the IR light becomes recognizable again by the light-emission recognizer <NUM>) (step S126/Yes), the information processing device <NUM> generates the trajectory image and performs control to display the trajectory image again (step S129).

In contrast, if the IR light is not recognizable (step S106/No) or if the IR light becomes unrecognizable at a position other than the user position (step S120/No) in the middle of recognizing the IR light (in the middle of drawing), the information processing device <NUM> displays the UI (guidance images 422a to 422c as illustrated in <FIG>) for guiding the operating tool <NUM> to the writable place (drawable area) in the same manner as in the embodiment described with reference to <FIG> at the writable place (step S112). The case of being unrecognizable at the position other than the user position is assumed to be a case where the IR light is out of the recognition area <NUM> (moved out of the angle of view) or a case where the IR light is shielded by a shielding object other than the user, for example, furniture or movable objects. As a result, the user is able to intuitively understand that it is not running out of battery of the operating tool <NUM> or a system trouble and it is possible to perform the drawing, and that it is possible to perform the drawing in the periphery where the guidance image is displayed. It is to be noted that, in a case where the shielding object has been recognized by object recognition or the like based on a captured image, or in a case where it has been recognized that shielding occurs due to any shielding object, the information processing device <NUM> may display a guidance image for giving notice of the shielding.

The information processing device <NUM> repeats the process illustrated in steps S106 to S112 and S120 to S <NUM> described above while the switch <NUM> of the pen tip continues to be pushed (step S132).

Subsequently, a modification example of the present embodiment will be described.

The guidance image according to the present embodiment may further indicate a vertical direction of the drawable area in accordance with the installation position of the projector <NUM> or the camera <NUM>.

<FIG> is a diagram illustrating the vertical direction of the drawable area according to the modification example of the present embodiment. As illustrated in <FIG>, a case is assumed where, for example, a top surface of a table <NUM> is the projection area and the user is able to draw from any direction.

In the periphery of the table <NUM>, the projector <NUM> that projects a trajectory image onto the top surface of the table <NUM> and the camera <NUM> that images IR light emission (drawing) of the operating tools 400a and 400b are installed.

Here, depending on a positional relationship of the projector <NUM> and the camera <NUM> to the user, there may be a place (i.e., an undesirable, disadvantageous position) at which shielding caused by the user easily occurs.

For example, as illustrated in <FIG>, no shielding occurs at a standing position of a user U1, but at a standing position of a user U2, the shield area <NUM> is generated by shielding the projection of the projector <NUM> that performs projection from behind the user U2, and the shield area <NUM> is generated by shielding the imaging of the camera <NUM>. It is not possible to perform drawing (recognition, projection) in the shield area <NUM> and the shield area <NUM>, and has a disadvantage that a size of the drawable area <NUM> decreases.

Therefore, in the present modification example, in order to clearly indicate a desirable standing position (drawing direction) to the user, the information processing device <NUM> may display a guidance image in which the vertical direction is indicated in the case where the shielding occurs (for example, a case where an operating tool 400b used by the user U2 is being operated but the light emission is unrecognizable, or the like). The vertical direction may be determined according to the positions of the projector <NUM> and the camera <NUM>. That is, the guidance image that indicates the vertical direction is to guide the user to the desirable and advantageous standing position (direction) corresponding to the positions of the projector <NUM> and the camera <NUM>.

<FIG> is a diagram illustrating an example of the guidance image that clearly indicates the vertical direction of the drawable area according to the present modification example. For example, as illustrated in <FIG>, a frame-like guidance image <NUM> including illustrations such as animals that indicate the top and bottom may be displayed. Further, other illustrations, icons, guide displays (characters), and the like that indicate the top and bottom may be displayed. This allows the user to intuitively understand the preferred standing position (drawing direction).

In the embodiment described above, the case where the projector <NUM> and the camera <NUM> are installed apart from each other has been described, but the present embodiment is not limited thereto. The projector <NUM> and the camera <NUM> may be installed such that the optical axis centers thereof are as close as possible or coaxial with each other, thereby regarding the shielding of the projector <NUM> as the shielding of the camera <NUM> and the guidance display may be performed.

That is, in the case where the projector <NUM> and the camera <NUM> are installed such that the optical axis centers thereof are as close as possible or coaxial with each other, the place of the shield area <NUM> in the case where the user shields the imaging of the camera <NUM> is expected to be substantially the same as the place of the shield area <NUM> in the case where the user shields the projection of the projector <NUM>. Thus, the information processing device <NUM> may regard the place of the user's shadow (shield area <NUM> of the projector <NUM>) as the shield area <NUM> of the camera <NUM>, and, in a case where the IR light has become unrecognizable in the vicinity thereof, notice may be given that the user is the factor of non-recognizability as illustrated in <FIG>.

A preferred embodiment(s) of the present disclosure has/have been described above in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such an embodiment(s). It is apparent that a person having ordinary skill in the art of the present disclosure can arrive at various alterations and modifications within the scope of the technical idea described in the appended claims, and it is understood that such alterations and modifications naturally fall within the technical scope of the present disclosure.

Claim 1:
An information processing device (<NUM>) comprising
a controller configured to
perform a recognition process of recognizing, on a basis of a captured image obtained by an imaging device (<NUM>) imaging a projection area (<NUM>), a position of an operating tool (<NUM>) inside the projection area (<NUM>), and
perform an operation detection process of detecting, on a basis of sensor data outputted by a sensor (<NUM>) included in the operating tool (<NUM>), that an operation is performed on the operating tool (<NUM>), and characterized in that,
in a case where the operation performed on the operating tool (<NUM>) is detected by the operation detection process and the position of the operating tool (<NUM>) is not recognized by the recognition process, the controller is configured to perform a projection control process of causing a projection device (<NUM>) to project an image for guiding the operating tool (<NUM>) to a position which is inside the projection area (<NUM>) and where the position of the operating tool (<NUM>) is recognizable by the recognition process.