NON-TRANSITORY STORAGE MEDIUM ENCODED WITH COMPUTER READABLE INFORMATION PROCESSING PROGRAM, INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, AND INFORMATION PROCESSING METHOD

A non-transitory storage medium encoded with a computer readable information processing program is provided. The information processing program, executed by a processing apparatus that is adapted to access a display unit and an input unit, causes the processing apparatus to perform functionality that includes causing a captured image captured by a virtual camera located in a virtual space to be displayed on the display unit, receiving an indicated position on the captured image from the input unit, calculating a position in the virtual space corresponding to the indicated position, and updating the captured image to a state in which ranges close to and distant from the virtual camera with respect to a range proximate to the calculated position are out of focus.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

The present embodiment will be described below in detail with reference to the drawings. It is noted that, in the drawings, the same or corresponding portions have the same reference characters allotted, and detailed description thereof will not be repeated.

First, a system configuration of an information processing system according to an embodiment will be described.

Referring toFIG. 1, an information processing system1according to the present embodiment includes a processing apparatus100, a display device130and an input device140. It is assumed that information processing system1shown inFIG. 1is typically mounted as a personal computer, a console-type game device, a set top box, or the like. Processing apparatus100is an essential part of implementing which executes various types of processing according to the present embodiment. Processing apparatus100executes various types of processing including information processing in response to a command given by a user operating input device140, and outputs a result thus obtained (a captured image) to display device130.

Display device130can be implemented by any device that can display a captured image in accordance with a signal (command) from processing apparatus100. Typically, a liquid crystal display, a plasma display, an organic electroluminescence display, or the like can be adopted as display device130. As input device140, various operation buttons, a keyboard, a touch panel, a mouse, an operating stick, or the like can be adopted.

Processing apparatus100includes, as main hardware, a CPU (Central Processing Unit)102, a GPU (Graphical Processing Unit)104, a RAM (Random Access Memory)106, a flash memory108, an output interface110, a communication interface112, and an input interface114. These components are connected to one another via a bus116.

CPU102is an essential part of processing which executes various programs. GPU104executes processing of producing a captured image as will be described later, in cooperation with CPU102. RAM106functions as a working memory which stores data, parameters, and the like necessary for execution of a program in CPU102and GPU104. Flash memory108stores in a nonvolatile manner an information processing program120executed in CPU102, various parameters set up a user, and the like.

Output interface110outputs a video signal or the like to display device130in accordance with an internal command from CPU102and/or GPU104. Communication interface112sends/receives data to/from another device by wire or wirelessly. Input interface114receives an operation signal from input device140for output to CPU102.

Although information processing system1shown inFIG. 1is illustrated as being configured such that display device130and input device140are provided separately from processing apparatus100, they may be configured integrally. An information processing system2shown inFIG. 2includes a processing apparatus100# having a display function and an input function. It is noted that, among the components shown inFIG. 2, components substantially the same as those shown inFIG. 1are represented by the same reference numbers, and similar components are represented by the same reference numbers followed by #. It is assumed that information processing system2shown inFIG. 2is typically implemented as a Smartphone, a PDA (Personal Digital Assistance), a mobile phone, a mobile gaming device, a digital camera, or the like. An external interface122is capable of reading, from non-transitory recording medium124of any type, data such as a program stored therein, and writing various types of data stored in flash memory108# or the like to non-transitory recording medium124.

The configurations shown inFIGS. 1 and 2are not limitations, but any system or apparatus that is configured to include an essential part of processing, a display unit and an input unit may be adopted. That is, the processing apparatus according to the present embodiment may be implemented in any form that is adapted to access a display unit and an input unit.

As described above, information processing system1according to the present embodiment includes display device130, input device140and processing apparatus100. Alternatively, according to another embodiment, an information processing apparatus (processing apparatus100#) that is adapted to access a display unit and an input unit is provided.

Furthermore, the present embodiment is embodied as information processing program120executed by the processing apparatus that is adapted to access a display unit and an input unit and as an information processing method executed in the processing apparatus that is adapted to access a display unit and an input unit.

For ease of description, basically, exemplary processing in the case where an information processing program is executed in information processing system1shown inFIG. 1will be described.

<B. Summary of Processing>

A summary of processing related to information processing according to the present embodiment will be given below.

As shown inFIG. 3, a plurality of objects OBJ1, OBJ2 are located in a virtual space200, and a captured image obtained by capturing an image of these objects by a virtual camera210is produced. The sight line direction of virtual camera210will be called a camera direction (optical axis direction) AX. A position in this camera direction will also be called a depth direction.

FIG. 4Ashows an example of a captured image produced by virtual camera210shown inFIG. 3capturing an image of the objects in virtual space200. In the captured image shown inFIG. 4A, object OBJ1 located at a position closer to virtual camera210is drawn larger in virtual space200, and object OBJ2 located at a position more distant from virtual camera210is drawn smaller. Basically, as for the captured image shown inFIG. 4A, the state of focus is identical in the depth direction. That is, all the objects in the field of view of virtual camera210are drawn in focus.

The information processing according to the present embodiment provides processing that can give a user a sense of realism such as that obtained when capturing an image by a real camera. Specifically, when capturing an image by a real camera, a subject to be subjected to image capturing is indicated, and an optical system is adjusted such that the indicated subject comes into focus. Since a camera has a depth of field as an optical property, a subject in focus is seen more sharply, while a subject out of focus is seen indistinctly.

Such actual focus adjustment is considered. In such a case where objects are located at a plurality of different depth positions, when one of the objects is brought into focus, another object is naturally seen indistinctly. The present embodiment gives a user a sense of realism such as that obtained when capturing an image of a subject by a real camera.

For example, as shown inFIG. 4B, when a target TRG1 is set as object OBJ1, object OBJ1 is drawn in focus, and object OBJ2 is drawn out of focus. At this time, the range to be drawn in focus is determined based on a depth position FP1 (FIG. 3) corresponding to object OBJ1.

As shown inFIG. 4C, when a target TRG2 is set as object OBJ2, object OBJ2 is drawn in focus, and object OBJ1 is drawn out of focus. At this time, the range to be drawn in focus is determined based on a depth position FP2 (FIG. 3) corresponding to object OBJ2.

In this manner, the range proximate to the depth position corresponding to an indicated object is drawn in focus, that is, sharply, and the remaining range is drawn out of focus. In the present embodiment, the range shown in focus and the range shown out of focus are dynamically changed in accordance with a user's instruction. Execution of such information processing can give a user a sense of realism such as that obtained when capturing an image by a real camera.

Referring toFIG. 5, a procedure of information processing according to the present embodiment will be described below.

Each step shown inFIG. 5is typically achieved by CPU102(FIG. 1) of processing apparatus100executing information processing program120. Referring toFIG. 5, CPU102of processing apparatus100receives information on the objects and a virtual camera located in a virtual space (step S1). Subsequently, CPU102causes a virtual camera to virtually capture an image of an object in the virtual space to produce (render) a captured image (step S2), and causes the produced captured image to be displayed on display device130(step S3). CPU102then receives an indicated position on the captured image from input device140(step S4). Typically, as shown inFIG. 4A, a position is indicated on the displayed captured image.

When a position is indicated by a user, CPU102calculates the depth position in the virtual space corresponding to the indicated position (step S5). When this depth position is calculated, CPU102updates the captured image such that the ranges close to and distant from the virtual camera with respect to a range proximate to the calculated depth position are out of focus (step S6). As seen from the user, in the captured image displayed previously, (part of) an object corresponding to the range (in the depth direction) distant from the range proximate to the calculated depth position is displayed out of focus.

Thereafter, CPU102determines whether or not termination of display processing has been instructed (step S7). When termination of display processing has not been instructed (NO in step S7), processing of and after step S1is repeated. That is, CPU102repeatedly receives indicated positions, and repeatedly calculates a corresponding position in the virtual space every time an indicated position is received.

On the other hand, when termination of display processing has been instructed (YES in step S7), processing is terminated.

Referring toFIG. 6, a functional configuration for processing apparatus100according to the present embodiment to achieve the above-described information processing will be described below.

Processing apparatus100includes, as its control configuration, an interface processing unit150, a position calculation unit160, a rendering unit170, and a data storage unit180. Interface processing unit150, position calculation unit160and rendering unit170shown inFIG. 6are typically implemented by CPU102(FIG. 1) of processing apparatus100executing information processing program120.

Interface processing unit150causes a captured image captured by the virtual camera located in the virtual space to be displayed on the display unit, and receives an indicated position on the captured image from the input unit. More specifically, interface processing unit150includes a display control unit152causing a captured image to be displayed on display device130or the like, and an instruction receiving unit154receiving an operation input from a user. Instruction receiving unit154outputs information on a position operation which is a user's instruction on the captured image, to position calculation unit160.

Position calculation unit160calculates the depth position in the virtual space corresponding to the indicated position. More specifically, position calculation unit160calculates the depth position corresponding to the user's indicated position from information on the objects and the virtual camera in the virtual space or the like, in response to a position operation through display control unit152.

Rendering unit170produces a captured image obtained by rendering (virtually capturing an image) in the virtual space with reference to virtual space definition data182, object definition data184, virtual camera definition data186, and the like stored in data storage unit180. Upon receipt of information on the depth position from position calculation unit160, rendering unit170updates the captured image such that the ranges close to and distant from the virtual camera with respect to the range proximate to the calculated depth position are out of focus. Rendering unit170has a defocusing function172. This defocusing function172achieves drawing out of focus.

Data storage unit180holds virtual space definition data182, object definition data184and virtual camera definition data186. Virtual space definition data182includes various set values concerning the virtual space and the like. Object definition data184includes various set values concerning objects located in the virtual space and the like. Virtual camera definition data186includes various set values concerning the virtual camera located in the virtual space and the like. The contents of object definition data184and/or virtual camera definition data186may be appropriately updated along with the progress of related information processing (typically, game processing).

Hereinafter, more detailed processing in the main steps shown inFIG. 5and a corresponding functional module shownFIG. 6will be described.

<E. Calculation of Depth Position by Position Calculation Unit160>

As described above, when a certain position is indicated from input device140, the depth position in the virtual space corresponding to that indicated position is calculated. Referring toFIGS. 7A and 7B, this processing of calculating the depth position will be described.

FIG. 7Ashows the case of determining the depth position corresponding to an indicated position, andFIG. 7Bshows the case of determining the depth position corresponding to an indicated region.

Virtual camera210located in the virtual space virtually captures an image of objects included in a view volume250in accordance with its angle of view to produce a captured image.

FIG. 7Ashows processing in the case where virtual camera210virtually captures an image of in the virtual space. Suppose that, with a captured image being displayed on display device130, a user indicates any position (selected position230) on the captured image being displayed. Position calculation unit160obtains a coordinate corresponding to selected position230on the captured image, and causes virtual camera210to emit a virtual control ray (hereinafter also called a “ray”)240based on the obtained coordinate. This emission angle of ray240is determined depending on the coordinate of selected position230on the captured image and the captured image as well as the angle of view of virtual camera210. Position calculation unit160then determines whether or not ray240intersects (hits) some object (or some geometry) located in the virtual space.

In the case where ray240hits some object (or some geometry), position calculation unit160calculates a coordinate where the hit has been made, and calculates the depth position of the calculated coordinate. Alternatively, a coordinate representing the hit object (e.g., a central coordinate or a coordinate of center of gravity of the object) is calculated. This depth position calculated corresponds to the depth position in the virtual space at the indicated position.

On the other hand, in the case where ray240does not hit any object (or any geometry), position calculation unit160outputs a predetermined depth position as the depth position in the virtual space at the indicated position.

It is noted that, even if ray240hits some object (or some geometry), when a point where the hit has been made is not included in a predetermined range, the predetermined depth position may be output as the depth position in the virtual space at the indicated position.

In this manner, the processing of calculating the depth position in the virtual space corresponding to the indicated position includes processing of calculating a position from the coordinate in the virtual space corresponding to the indicated position (selected position230). That is, the depth position corresponding to one spot (point) indicated by the user is determined.

It is noted that, considering a real camera, a region defined in the finder shall be a region to be brought into focus in many cases. When imitating such a real camera, processing in which the user indicates a region to be brought into focus on the captured image being displayed and a corresponding depth position is determined based on this indicated region may be preferable.

FIG. 7Bshows exemplary processing of calculating the depth position based on a region in the virtual space corresponding to an indicated position. More specifically, suppose that, with the captured image being displayed on display device130, the user instructs a region (selected region232) centering on any position of the captured image being displayed. Position calculation unit160obtains coordinates of vertices that define selected region232on the captured image, and causes virtual camera210to emit a plurality of rays240-1to240-N based on the obtained coordinates, respectively. The emission angle of rays240-1to240-N is determined depending on the coordinates on the captured image of selected region232and the captured image as well as the angle of view of virtual camera210. Position calculation unit160determines whether or not each of rays240-1to240-N intersects (hits) some object (or some geometry) located in the virtual space.

Position calculation unit160extracts rays among rays240-1to240-N having hit some object (or some geometry), and calculates (basically, a plurality of) coordinates where each of the extracted rays has made the hit. Position calculation unit160calculates the depth position based on the calculated coordinates. That is, the processing of calculating the depth position in the virtual space corresponding to the indicated position includes processing of determining coordinates from a plurality of coordinates included in the region in the virtual space. That is, a plurality of depth positions corresponding to a plurality of spots (points) included in the region indicated by the user are extracted, and from among them, a representative depth position is determined.

A representative value of the depth position may be determined by performing various types of statistical processing on the plurality of depth positions thus extracted. As an example of such statistical processing, processing of determining an average value, a medium value, a highest frequency value, or the like as a representative value is conceivable. Of course, the statistical processing is not limited to these enumerated types of processing, but any statistical processing can be adopted.

That is, the processing of calculating the depth position in the virtual space corresponding to the indicated position includes processing of performing statistical processing on a plurality of coordinates included in a region in the virtual space, thereby determining a single coordinate.

It is noted that, if none of plurality of rays240-1to240-N hits any object (or any geometry), position calculation unit160outputs a predetermined depth position as the depth position in the virtual space at the indicated position. Alternatively, when the calculated depth position is not included in the predetermined range, the predetermined depth position may be output as the depth position in the virtual space at the indicated position.

When calculating the depth position, processing of restricting the depth position calculated, namely, processing of calculating the position in a predetermined region in the virtual space may be included. In other words, a corresponding depth position may be clamped.

Through the processing as described above, the corresponding depth position in the virtual space, that is, a reference position for determining a range to be updated to be out of focus is calculated.

It is noted that, once the corresponding depth position in the virtual space is calculated, even if the image capturing direction of virtual camera210is changed, the depth determined at the previously indicated position may be kept in focus. That is, even if the image capturing direction of virtual camera210is changed, the position corresponding to the indicated position may be held. By adopting such processing, the depth position in focus can be prevented from changing unintentionally from the depth position corresponding to the previously indicated position even though the user has not instructed a focusing operation. For example, even if a subject has come out of the field of view of virtual camera210in such a case where virtual camera210is moved in the virtual space, focus established on the subject can be prevented from being changed unintentionally.

<F. Processing of Updating to be Out of Focus by Rendering Unit170>

Processing of updating to defocusing function172mounted on rendering unit170will be described below.

Referring toFIG. 8, rendering unit170produces, as defocusing function172, a clarified image174produced by virtually capturing an image of objects in the virtual space in substantially identical focus in the depth direction and a defocused image176produced by virtually capturing an image of the objects in the virtual space out of focus. Rendering unit170then uses a mixing ratio α(x, y) at each pixel position determined by a method as will be described later to mix, for each pixel, a corresponding pixel value of clarified image174and a corresponding pixel value of defocused image176, thereby producing a target captured image178

That is, the pixel value at each pixel position (x, y) is calculated in accordance with Expression (1) indicated below.

Pixel value (x,y)=pixel value (x,y) of clarified image 174×α(x,y)+pixel value (x,y) of defocused image 176×(1−α(x,y))  (1)

Mixing ratio α(x, y) is dynamically determined depending on the corresponding depth position of the pixel position (x, y) in the virtual space. That is, the defocusing degree at each depth position is determined based on a defocusing degree profile in the depth direction as will be described later.

The defocusing degree profile produced by the information processing according to the present embodiment will be described below. In the present embodiment, the range of the depth position updated to be out of focus is adjusted in association with a parameter concerning the virtual camera. In the following description, the relationship between the distance from the virtual camera (depth position) and the defocusing degree will be called “a defocusing degree profile.” That is, in the processing of drawing a captured image according to the present embodiment, the defocusing degree is varied depending on the distance from virtual camera210to a calculated position (reference depth distance).

Referring toFIGS. 9A,9B,10A, and10B, processing of dynamically varying the defocusing degree profile will be described.

Basically, a range except a range centering on the depth position corresponding to the indicated position is updated to be out of focus. That is, in the processing of drawing a captured image, as compared with the drawing state of a proximate range centering on the depth position, the remaining range is drawn out of focus.

In the present embodiment, the range in the depth direction in which a drawing is to be made in focus (sharply) and the range in the depth direction to be updated to be out of focus are determined at least depending on the depth position corresponding to the indicated position and/or the angle of view of virtual camera210.

As an example, as shown inFIGS. 9A,9B,10A, and10B, a range300in which a drawing is to be made in focus is set centering on the corresponding depth position. The defocusing degree is continuously varied in ranges continuous with this range300. Such a defocusing degree profile is defined by a reference depth position310, a backside defocusing start position312, a backside defocusing completion position314, a front side defocusing start position316, and a front side defocusing completion position318. Here, reference depth position310corresponds to the depth position in the virtual space corresponding to an indicated position. The distance from virtual camera210to reference depth position310will be called “a reference depth distance.”

Backside defocusing start position312, backside defocusing completion position314, front side defocusing start position316, and front side defocusing completion position318are calculated in accordance with Expressions (2) to (5) indicated below, respectively, for example.

Here, A indicates a predetermined offset value, β indicates a predetermined first correction value, and γ(θ) indicates a second correction value depending on the angle of view of virtual camera210.

FIGS. 9A and 9Bshow an exemplary change in the defocusing degree profile along with a change in the distance (reference depth distance) from virtual camera210to reference depth position310.FIG. 9Ashows an example of backside defocusing start position312, backside defocusing completion position314, front side defocusing start position316, and front side defocusing completion position318at a reference depth distance L1.FIG. 9Bshows an example of backside defocusing start position312, backside defocusing completion position314, front side defocusing start position316, and front side defocusing completion position318at a reference depth distance L2 (>L1).

As is clear from the comparison betweenFIGS. 9A and 9B, range300to be updated into focus becomes wider as the reference depth distance becomes longer. The ranges in which a change from the start of defocusing to the completion of defocusing is made also become wider as the reference depth distance becomes longer. That is, the proximate ranges are made larger as the distance from virtual camera210to the calculated position becomes longer.

By thus causing the ranges to be updated to be out of focus to depend on the reference depth distance, a natural display closer to the state of capturing an image by a real camera can be achieved.

In this manner, in the processing of drawing a captured image according to the present embodiment, the proximate range to be updated into focus is determined in accordance with the distance (reference depth distance) from virtual camera210to the calculated position (reference depth position310). In other words, in the processing of drawing a captured image according to the present embodiment, the defocusing degree is gradually increased away from the proximate range to be updated into focus.

This proximate range is determined depending on the optical characteristics set for virtual camera210. As such optical characteristics, various types of parameters can be adopted. Typically, the proximate range is determined depending on the depth of field set for virtual camera210.

In this manner, by gradually increasing the defocusing degree away from the proximate range to be updated into focus, that is, by decreasing the width of the proximate range when close to virtual camera210and increasing the width of the proximate range when distant from virtual camera210, a natural display closer to the state of capturing an image by a real camera can be achieved.

FIGS. 10A and 10Bshow an exemplary change in the defocusing degree profile along with a change in the angle of view of virtual camera210.FIG. 10Ashows an example of backside defocusing start position312, backside defocusing completion position314, front side defocusing start position316, and front side defocusing completion position318in the case where the angle of view of virtual camera210is θ1.FIG. 10Bshows an example of backside defocusing start position312, backside defocusing completion position314, front side defocusing start position316, and front side defocusing completion position318in the case where the angle of view of virtual camera210is θ2 (>θ1).

As is clear from the comparison betweenFIGS. 10A and 10B, range300to be updated into focus becomes wider as the angle of view of virtual camera210becomes larger. On the other hand, the ranges in which a change from the start of defocusing to the completion of defocusing is made becomes smaller as the angle of view of virtual camera210becomes larger. That is, the change from the start of defocusing to the completion of defocusing becomes sharper as the angle of view of virtual camera210becomes larger.

In this manner, by causing the range to be updated to be out of focus to depend on the angle of view of virtual camera210, that is, by increasing the range to be brought into focus with a change in the angle of view of virtual camera210, a natural display closer to the state of capturing an image by a real camera can be achieved.

It is noted that, althoughFIGS. 9A,9B,10A, and10B show the examples in which the defocusing degree is linearly changed from the start of defocusing to the completion of defocusing depending on the distance from virtual camera210, they are not limitations, but any changing characteristic can be adopted.

Referring toFIGS. 11A to 11D, a variation of the defocusing degree profile according to the present embodiment will be described. As shown inFIG. 11A, the defocusing degree may be increased in proportion to the distance from virtual camera210, or as shown inFIGS. 11B and 11C, the defocusing degree may be increased nonlinearly relative to the distance from virtual camera210. Alternatively, as shown inFIG. 11D, a drawing may be made with the defocusing degree set at zero in a certain range, and when the degree exceeds that range, the defocusing degree may be maximized. That is, a changing profile of the defocusing degree in which no intermediate defocusing degree exists may be adopted.

As described above, in the processing of drawing a captured image according to the present embodiment, the changing profile of the defocusing degree is determined depending on at least one of the distance (reference depth distance) from virtual camera210to the calculated position (reference depth position310) and the angle of view of virtual camera210.

More specifically, as shown inFIGS. 9A and 9B, the changing profile of the defocusing degree is determined such that the amount of change in the defocusing degree relative to the distance decreases as the distance (reference depth distance) from virtual camera210to the calculated position becomes longer.

As shown inFIGS. 10A and 10B, the changing profile of the defocusing degree is determined such that the amount of change relative to the distance increases as the angle of view of virtual camera210becomes larger.

By thus changing the defocusing degree in accordance with the reference depth position, settings of virtual camera210and the like, a natural display closer to the state of capturing an image by a real camera can be achieved.

Although the above-described embodiment illustrates the processing of changing the defocusing degree profile and the like depending on the distance (reference depth distance) from virtual camera210to the calculated position and/or the angle of view of virtual camera210, the defocusing degree profile may be dynamically changed depending on a parameter different from them.

For example, when the information processing according to the present embodiment is applied to an application in which the position of virtual camera210in the virtual space is changed with time, that is, virtual camera210is moved, the defocusing degree profile may be dynamically changed in accordance with the moving speed of virtual camera210. More specifically, a sense of speed can be given to a user by narrowing the range in the depth direction in which drawing is made in focus (sharply) as the moving speed of virtual camera210becomes higher.

Furthermore, the defocusing degree profile may be dynamically changed depending on the brightness in the virtual space or the like.

As described above, according to the present embodiment, a sense of realism such as that obtained when capturing an image by a real camera can be given to a user.

While certain example systems, methods, devices and apparatuses have been described herein, it is to be understood that the appended claims are not to be limited to the systems and methods, devices and apparatuses disclosed, but on the contrary, and are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.