Patent Description:
There is a system in which an image obtained by capturing a state of a live performance performed in a live venue is distributed to a device on a viewer side that is remote, and the image of the live performance is reproduced by the device on the viewer side. For example, Patent Literature <NUM> discloses a technique of controlling stage lighting provided in a live venue based on evaluation information transmitted from a viewer side to a distributor side. Patent Literature <NUM> discloses a technique of controlling a lighting device provided on a viewer side based on lighting information of a live venue.

<CIT> relates to an event presentation control system, an event notification system, and a program used for an event. A specific participant is specified based on action information and a special character is provided to the specific participant in order to improve the excitement in the event.

<CIT> relates to a system of synchronizing lighting effect control signals and patterns for controlling interactive lighting effect devices in order to allow efficient lighting effect pattern generation at a remote location.

<CIT> relates to a video conferencing system which creates the illusion of being in the same room with a remotely located person.

In such a system, there is a demand for a mechanism that causes a viewer who remotely views an image of a live performance to feel presence of the live performance.

The present invention has been made in view of such circumstances, and an object thereof is to provide an image display system, a display control method, a light emission control method and a program capable of causing a viewer who remotely views an image of a live performance to feel presence of the live performance.

The present invention is solved in the appended independent claims.

In order to solve the above-mentioned problems, according to an aspect of the present invention, there is provided a display control method executed by a computer. The display control method includes acquiring image information by an image information acquisition unit, the image information indicating a target image in which a situation of a live performance being performed in a live venue is captured; and displaying, by a display device, a virtual space in which the target image corresponding to the image information is displayed; and setting, by a display control unit (<NUM>), a virtual lighting as a light source in the virtual space; controlling, by a display control unit, light to be emitted by a virtual lighting device provided in the virtual space based on state information indicating a state of a target space in the target image, wherein the target space is a live space in which the live performance is present; and by the display control unit, generating a display control signal and outputting the generated display control signal to control the virtual space, to be displayed by the display device.

According to another aspect of the present invention, there is provided an image display system. The image display system includes an image information acquisition unit configured to acquire image information, the image information indicating a target image in which a situation of a live performance being performed in a live venue is captured; and a display device configured to display a virtual space in which the target image corresponding to the image information is displayed, and a display control unit configured to: set a virtual lighting as a light source in the virtual space; control light to be emitted by the virtual lighting based on state information indicating a state of a target space in the target image, wherein the target space is a live space in which the live performance is present; generate a display control signal and output the generated display control signal to control the virtual space, to be displayed by the display device.

According to the present invention, a viewer who remotely views an image of a live performance can feel presence of the live performance.

<FIG> is a diagram showing an overview of an image display system <NUM> according to an embodiment. The image display system <NUM> is a system that distributes a situation of a live performance being performed in a live venue P1 to a remote location P2. The remote location P2 is, for example, a home of a user remote from the live venue P1.

In the image display system <NUM>, for example, an image signal, a sound signal, a lighting signal and the like in the live venue P1 are acquired as the situation of the live performance, and these signals are distributed to a control device <NUM>. The image signal is, for example, a signal of an image indicating a video or the like captured in the live venue P1. The sound signal is, for example, a signal of sound indicating sound collected in the live venue P1. The lighting signal is, for example, a signal for controlling various kinds of stage lighting provided in the live venue P1.

In the image display system <NUM>, the control device <NUM> is, for example, a computer device provided at the remote location P2. At the remote location P2, one or more lighting devices <NUM> controlled by the control device <NUM> are provided. For example, the lighting device <NUM> is arranged according to a configuration of the stage lighting in the live venue. Although it is not necessary for the lighting device <NUM> to completely match the configuration of the stage lighting in the live venue, it is desirable that the lighting device <NUM> has a configuration the same as the configuration of the stage lighting in the live venue as much as possible and can create a lighting effect the same as that of the live venue.

The control device <NUM> receives the image signal, the sound signal and the lighting signal in the live venue P1. The control device <NUM> generates a lighting control signal, a display control signal and a sound control signal based on the received signals. The lighting control signal is a control signal for controlling light to be emitted by the lighting device <NUM> at the remote location P2, and is, for example, a digital multiplex <NUM> (DMX512) signal. The display control signal is a control signal for controlling an image to be displayed on a display device (a display device <NUM> in <FIG>) at the remote location P2. The sound control signal is a control signal for controlling sound to be output from a speaker device (a speaker device <NUM> in <FIG>) at the remote location P2. <FIG> is a block diagram showing an example of a configuration of the image display system <NUM> according to the embodiment. A specific method for generating the lighting control signal and the like by the control device <NUM> will be described in detail later.

The control device <NUM> outputs the generated lighting control signal to the lighting device <NUM>. Thereby, the lighting device <NUM> emits the light according to the lighting control signal. The control device <NUM> outputs the generated display control signal to the display device <NUM>. Thereby, the display device <NUM> displays the image corresponding to the display control signal. The control device <NUM> outputs the generated sound control signal to the speaker device <NUM>. Thereby, the speaker device <NUM> outputs the sound corresponding to the sound control signal. Thereby, the image display system <NUM> can reproduce the situation of the live venue P1 at the remote location P2.

In the present embodiment, the control device <NUM> causes the display device <NUM> to display a virtual space. The virtual space is a virtual space simulating the live venue, and is, for example, a space in which a virtual screen or a virtual stage is provided at a center and virtual spectator seats are provided around the virtual screen or the virtual stage. The control device <NUM> displays, for example, the video captured in the live venue P1 on the screen provided in the virtual space. Examples of the display device <NUM> include a PC monitor and a head mounted display (abbreviated as HMD). Specific examples of the display device <NUM> include HoloLens (registered trademark) of Microsoft (registered trademark) ([searched on January <NUM>, <NUM>] with reference to the Internet <URL: https://www. com/en-us/hololens>).

The control device <NUM> displays the virtual space according to a movement of the user or the like. Thereby, the user can view the live performance while freely moving in the virtual space. The control device <NUM> receives, from an input device (an input device <NUM> in <FIG>) such as an HMD, an input signal indicating an actual line-of-sight direction of the user, the movement of the user viewing the screen displayed on the display device <NUM>, or the like. The control device <NUM> changes a viewing position of the user in the virtual space and a line-of-sight direction of the user in the virtual space according to the received signal, and causes the display device <NUM> to display the virtual space viewed in the line-of-sight direction from the changed viewing position in real time. Thereby, the image display system <NUM> can display the image of the virtual space according to the viewing position and the line-of-sight direction in the virtual space freely selected by the user. Therefore, the user can experience the situation of the live performance as if the user freely walks around the live venue P1 and views the live venue P1 from various positions.

The control device <NUM> controls the light to be emitted by the lighting device <NUM> according to the viewing position of the user in the virtual space. For example, when the viewing position of the user in the virtual space is in front of the stage, the control device <NUM> controls the light to be emitted by the lighting device <NUM> using the lighting signal corresponding to the stage lighting provided on a stage upper portion (or a stage floor surface). For example, when the viewing position of the user in the virtual space moves to a side of the stage, the light to be emitted by the lighting device <NUM> is controlled by using the lighting signal corresponding to the stage lighting which irradiates a stage with light from the side of the stage. Thereby, the image display system <NUM> can control light emission of the lighting device <NUM> according to a movement of an actual line of sight or the like of the user. Therefore, the user can freely walk around the live venue P1 and feel lighting provided in the live venue P1.

The control device <NUM> controls the light to be emitted by the lighting device <NUM> according to state information indicating a state of a live space. The state information is information indicating the state of the live venue in which the user viewing the image or the like of the live performance at the remote location P2 feels presence of the live performance. The state information is, for example, a state of light that can stimulate human senses in the live venue. Alternatively, the state information is information indicating the sound in the live venue. The state information is, for example, information for controlling the stage lighting provided in the live venue, for example, a DMX512 signal. The state information may be information indicating a representative color representing a color of the image in which the situation of the live performance is captured. The state information may be information indicating a color corresponding to a frequency of the sound collected in the live space. The state information may be information indicating an environment of the live venue, for example, information indicating a humidity, a smell, a temperature and the like in the live venue (see <FIG>).

Thereby, for example, when the lighting is turned off or a quiet ballad is played in the live venue, the control device <NUM> turns off lighting of the display device <NUM> to create a quiet atmosphere. When the lighting is turned on at once or a play is at a high volume in the live venue, the control device <NUM> makes the lighting of the display device <NUM> to be bright or to blink based on the state information, thereby creating an atmosphere in which the live performance is exciting. This enables the user to feel a sense of unity with the live venue P1.

When the play is at the high volume in the live venue, not only the lighting of the display device <NUM> may be made to be bright or to blink, but also a volume to be output from the speaker device <NUM> may be increased. In this case, for example, the control device <NUM> outputs a signal obtained by changing a volume or the like of the sound signal collected in the live space by an equalizer to the speaker device <NUM> as the sound control signal.

As described above, in the image display system <NUM>, a live video can be viewed from various positions in the virtual space, and the light to be emitted by the lighting device <NUM> is controlled according to the viewing position and the state information in the virtual space. Thereby, in the image display system <NUM>, the user can freely select the viewing position in the virtual space, and can feel the lighting provided in the live venue P1 according to the viewing position in the virtual space. Therefore, the user can experience the situation of the live performance from various positions as if the user freely walks around the live venue P1, and the light emission of the lighting device <NUM> changes according to the viewing position in the virtual space and a state of the live performance, whereby the use can experience the presence of the live performance better.

As shown in <FIG>, the image display system <NUM> includes, for example, a device group on a live venue P1 side and a device group on a remote location P2 side. A distribution server <NUM> of the device group on the live venue P1 side and the control device <NUM> on the remote location P2 side are communicably connected to each other via a communication network NW.

In the following, a case where the situation of the live performance is distributed in real time will be described as an example, but the present invention is not limited thereto. It is also possible to adopt a mode in which data or the like in which the situation of the live performance is recorded can be acquired on demand by using a file or the like. When the data is read using a file or the like, the control device <NUM> acquires the data from, for example, a memory (a storage unit <NUM> in <FIG> to be described later) provided in the control device <NUM>. In this case, a communication function of the control device <NUM> (a communication unit <NUM> in <FIG> to be described later) may be omitted.

In the following, a case where the image signal, the sound signal and the lighting signal in the live venue P1 are distributed from one distribution server <NUM> will be described as an example, but the present invention is not limited thereto. The image signal, the sound signal and the lighting signal may be distributed from separate servers, or may be acquired in a file format.

The device group on the live venue P1 side is a device group that generates and distributes each of the image signal, the sound signal and the lighting signal in the live venue P1. The device group on the live venue P1 side includes, for example, an imaging device <NUM>, a sound collecting device <NUM>, a lighting control device <NUM> and the distribution server <NUM>. The imaging device <NUM> captures the image of the live performance being performed in the live venue P1, and outputs a captured image signal to the distribution server <NUM>. The sound collecting device <NUM> collects the sound in the live venue P1 and outputs a collected sound signal to the distribution server <NUM>. The lighting control device <NUM> controls the lighting provided in the live venue P1 using the lighting signal, and outputs the lighting signal to the distribution server <NUM>. The distribution server <NUM> transmits each of the image signal, the sound signal and the lighting signal generated in the live venue P1 to the control device <NUM>.

For example, when the state information is information for controlling the stage lighting provided in the live venue P1, the state information is generated based on the lighting signal used by the lighting control device <NUM> for controlling the stage lighting. When the state information is information indicating the representative color representing the color of the image in which the situation of the live venue P1 is captured, the state information is generated based on the image signal of the image captured by the imaging device <NUM>. When the state information is information indicating the color corresponding to the frequency of the sound collected in the live venue P1, the state information is generated based on the sound signal of the sound collected by the sound collecting device <NUM>. When the state information is information indicating the environment of the live venue P1, the state information is generated based on the humidity, the smell, the temperature and the like measured in the live venue P1. In this case, information indicating the humidity, the smell, the temperature and the like measured in the live venue P1 is aggregated in the distribution server <NUM>. The distribution server <NUM> transmits the information indicating the humidity, the smell, the temperature and the like measured in the live venue P1 to the control device <NUM>. The state information may be generated in the live venue P1 or may be generated in the remote location P2. When the state information is generated in the live venue P1, for example, the imaging device <NUM> or the distribution server <NUM> generates the representative color of the image as the state information based on the image signal of the image in which the live venue P1 is captured. The distribution server <NUM> distributes the generated state information to the remote location P2. When the state information is generated in the remote location P2, the control device <NUM> receives the image signal of the image in which the live venue P1 is captured from the distribution server <NUM>, and generates the representative color of the image as the state information based on the received information.

The device group on the remote location P2 side is a device group that reproduces the situation of the live performance at the remote location P2 using each of the image signal, the sound signal and the lighting signal in the live venue P1. The device group on the remote location P2 side includes, for example, the control device <NUM>, the lighting device <NUM>, the display device <NUM>, the speaker device <NUM> and the input device <NUM>. The control device <NUM> receives each of the image signal, the sound signal and the lighting signal generated in the live venue P1 from the distribution server <NUM>. The control device <NUM> generates the lighting control signal, the display control signal and the sound control signal by using various signals received from the distribution server <NUM>. The control device <NUM> outputs the lighting control signal to the lighting device <NUM>. The lighting device <NUM> emits the light according to the lighting control signal. The control device <NUM> outputs the display control signal to the display device <NUM>. The display device <NUM> displays the image corresponding to the display control signal. The control device <NUM> outputs the sound control signal to the speaker device <NUM>. The speaker device <NUM> outputs the sound corresponding to the sound control signal. The input device <NUM> is attached to the user to detect the movement of the actual line of sight or the like of the user, and outputs a detection result to the control device <NUM> as input information. The control device <NUM> generates the display control signal indicating the virtual space viewed from the viewing position in the virtual space based on the input information, and outputs the generated display control signal to the display device <NUM>. The control device <NUM> generates the lighting control signal based on the viewing position in the virtual space based on the input information, and outputs the generated lighting control signal to lighting device <NUM>.

<FIG> is a block diagram showing an example of a configuration of the control device <NUM> according to the embodiment. The control device <NUM> includes, for example, the communication unit <NUM>, the storage unit <NUM> and a control unit <NUM>.

The communication unit <NUM> is implemented by, for example, a general-purpose integrated circuit (IC) for communication. The communication unit <NUM> communicates with the distribution server <NUM> via the communication network. The communication unit <NUM> communicates with the lighting device <NUM>, the display device <NUM>, the speaker device <NUM> and the input device <NUM>.

The storage unit <NUM> is implemented by, for example, a storage device such as a hard disk drive (HDD) or a flash memory (a storage device including a non-transitory storage medium), or a combination thereof. The storage unit <NUM> stores a program for implementing each component of the control device <NUM>, variables used when the program is executed, and various types of information. The storage unit <NUM> stores, for example, connection information <NUM> and state information <NUM>.

<FIG> is a table showing an example of a configuration of the connection information <NUM> according to the embodiment. The connection information <NUM> is information related to a device group connected to the control device <NUM>, and includes items such as lighting device information, display device information and speaker device information. The lighting device information is information on a lighting device connected to the control device <NUM>. The information on the lighting device is, for example, information indicating an installation position and a size of the lighting device, the number of lighting devices, a specification of each lighting device, a signal protocol of a control signal for controlling the lighting device, and the like. The display device information is information on a display device connected to the control device <NUM>. The information on the display device is, for example, information indicating an installation position and a size of the display device, a specification of the display device, a signal protocol of a control signal for controlling the display device, and the like. The speaker device information is information on a speaker device connected to the control device <NUM>. The information on the speaker device is, for example, information indicating an installation position and a size of the speaker device, the number of speaker devices, a specification of each speaker device, a signal protocol of a control signal for controlling the speaker device, and the like.

<FIG> is a table showing an example of a configuration of the state information <NUM> according to the embodiment. The state information <NUM> is generated based on distribution information distributed from the distribution server <NUM>. The state information <NUM> is information indicating the state of the live space, and includes, for example, items such as a state information identification ID, an input signal and a state information generation means. The state information identification ID is identification information for identifying the state information.

The input signal is a signal used when generating the state information specified by the state information identification ID. For example, when the state information is generated using a lighting signal, the "lighting signal" is indicated as the input signal. When the state information is generated using an image signal, the "image signal" is indicated as the input signal. When the state information is generated using a sound signal, the "sound signal" is indicated as the input signal. When the state information is generated using a signal indicating the humidity, "humidity information" is indicated as the input signal. When the state information is generated using a signal indicating the smell, "smell information" is indicated as the input signal. When the state information is generated using a signal indicating the temperature, "temperature information" is indicated as the input signal.

The state information generation means is information indicating specific processing of generating the state information specified by the state information identification ID. The state information identification ID is information indicated in the leftmost item of the state information <NUM>. For example, the state information specified by the state information identification ID (<NUM>) is generated by selecting the stage lighting according to the viewing position or the like in the virtual space. The state information specified by the state information identification ID (<NUM>) is generated by extracting the representative color from color information of the image. The state information specified by the state information identification ID (<NUM>) is generated by extracting the frequency of the sound. The state information specified by the state information identification ID (<NUM>) is generated using the humidity of the live venue. The state information specified by the state information identification ID (<NUM>) is generated using the smell of the live venue. The state information specified by the state information identification ID (<NUM>) is generated using the temperature of the live venue.

It should be noted that which state information is generated among a plurality of pieces of state information shown in an example of the state information <NUM> may be selected by an input operation of the user. In this case, the control device <NUM> displays a list of state information as shown in <FIG> on a display unit (not shown). The user operates, for example, an input device such as a mouse or a keyboard to designate which of the plurality of pieces of state information is to be generated.

When the lighting signal is not distributed or when the lighting signal is not embedded in the data distributed from the distribution server <NUM>, the state information may be automatically generated. In this case, for example, what kind of state information is generated based on which signal (for example, a sound signal or an image signal) is designated by the input operation or the like of the user.

The present invention is not limited to a case where only one of the plurality of pieces of state information is selected, and the plurality of pieces of state information may be designated to be generated at the same time.

Returning to description of <FIG>, the control unit <NUM> is configured to be implemented by, for example, a processor such as a central processing unit (CPU) or a digital signal processor (DSP) included in the control device <NUM> executing a program stored in the storage unit <NUM>. Alternatively, all or some of the above-described components in the control device <NUM> may be implemented by dedicated hardware such as a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

The control unit <NUM> includes, for example, a first acquisition unit <NUM>, a second acquisition unit <NUM>, a generation unit <NUM>, a lighting control unit <NUM>, a display control unit <NUM>, a sound control unit <NUM> and a device control unit <NUM>. The first acquisition unit <NUM> acquires distribution information. The distribution information is information indicating each of the image signal, the sound signal and the lighting signal in the live venue P1 transmitted from the distribution server <NUM> to the control device <NUM>. The first acquisition unit <NUM> outputs the acquired distribution information to the generation unit <NUM>, the display control unit <NUM> and the sound control unit <NUM>.

The second acquisition unit <NUM> acquires input information. The input information is information that is output from the input device <NUM> to the control device <NUM> and that indicates the movement of the actual line of sight or the like of the user. The second acquisition unit <NUM> outputs the acquired input information to the generation unit <NUM> and the display control unit <NUM>.

The generation unit <NUM> generates the state information <NUM> based on the distribution information distributed from the distribution server <NUM>, and stores the generated information as the state information <NUM> of the storage unit <NUM>. The generation unit <NUM> generates, for example, information indicating a state of light in the live venue as the state information <NUM>. In this case, the generation unit <NUM> generates the state information <NUM> based on the lighting signal, for example. For example, the generation unit <NUM> sets, as the state information <NUM>, a lighting signal of specific lighting of the stage lighting provided in the live venue, for example, the stage lighting in which a direction of light, an amount of light or the like changes with a movement of a person on the stage.

Alternatively, the generation unit <NUM> may extract information indicating a state of light in the live venue from an image signal, and may set a value corresponding to the extracted information as the state information <NUM>. For example, the generation unit <NUM> determines a color (representative color) representing an image (target image) corresponding to the image signal based on color information of the image, and sets a value corresponding to the color information of the determined representative color as the state information <NUM>. The generation unit <NUM> determines an RGB value of the representative color based on, for example, RGB values (an example of color information) of all pixels of the target image. For example, the generation unit <NUM> sets statistics such as an average value, a mode value, the maximum value and the minimum value in the RGB values of all the pixels as the RGB value of the representative color. Alternatively, the generation unit <NUM> may set a representative value of the RGB values of the pixels in a region obtained by dividing the target image into a plurality of regions as the RGB value of the representative color.

Alternatively, the generation unit <NUM> may extract information indicating sound in the live venue from a sound signal, and may set a value corresponding to the extracted information as the state information <NUM>. The generation unit <NUM> sets, for example, information indicating a strength, a height or the like of the sound signal as the state information <NUM>.

Specifically, the generation unit <NUM> calculates a signal intensity for each frequency of the sound constituting the sound signal, for example, by performing Fourier transform on the sound signal. The generation unit <NUM> sets, for example, a value corresponding to the largest signal intensity among all frequency bands constituting the sound as the state information <NUM>. Alternatively, the generation unit <NUM> may set a value corresponding to the smallest frequency among all the frequency bands constituting the sound as the state information <NUM>. The generation unit <NUM> may set a value corresponding to the largest frequency among all the frequency bands constituting the sound as the state information <NUM>. In this case, for example, the generation unit <NUM> sets, as the state information <NUM>, a value corresponding to which of a high frequency band, a medium frequency band and a low frequency band the largest frequency among all the frequency bands constituting the sound is. Alternatively, the generation unit <NUM> may set a value corresponding to a frequency of the largest signal intensity in all the frequency bands constituting the sound as the state information <NUM>. The generation unit <NUM> may set, for example, a value corresponding to the maximum value of the sound signal in a predetermined time interval as the state information <NUM>.

The generation unit <NUM> determines whether to control the lighting device <NUM> using the lighting information acquired from the first acquisition unit <NUM> or to control the lighting device <NUM> using the state information <NUM>. For example, the generation unit <NUM> switches information for controlling the lighting device <NUM> to the lighting information or the state information <NUM> according to an operation input by the user. Alternatively, the generation unit <NUM> may set information obtained by mixing the lighting information or the state information <NUM> as the information for controlling the lighting device <NUM>. The generation unit <NUM> outputs the information for controlling lighting device <NUM> to the lighting control unit <NUM>.

The lighting control unit <NUM> acquires the information for controlling the lighting device <NUM> from the generation unit <NUM>. The lighting control unit <NUM> generates a lighting control signal based on the information acquired from the generation unit <NUM>, and outputs the generated signal to the lighting device <NUM>. Thereby, the light emitted by the lighting device <NUM> is controlled. For example, the lighting control unit <NUM> generates the lighting control signal by converting the lighting signal of the specific stage lighting used for controlling the lighting device <NUM> in the distribution information into a control protocol of the lighting device <NUM> shown in the connection information <NUM> as necessary. The lighting control unit <NUM> outputs the generated lighting control signal to the lighting device <NUM>. When converting a control protocol of the lighting signal, the lighting control unit <NUM> performs conversion based on, for example, information (the control protocol of the lighting device <NUM>) stored in the connection information <NUM>.

For example, the lighting control unit <NUM> acquires an input signal indicating a movement of an actual line of sight or the like of the user from the generation unit <NUM> as the information for controlling the lighting device <NUM>. In this case, the lighting control unit <NUM> specifies a viewing position of the user in the virtual space based on the input information. The lighting control unit <NUM> changes the stage lighting used for controlling the lighting device <NUM> in the distribution information based on the specified viewing position and a line-of-sight direction in the virtual space. For example, the lighting control unit <NUM> selects the lighting signal of the stage lighting provided in vicinity of a position corresponding to the viewing position and the line-of-sight direction of the user in the virtual space in the live venue P1 as a signal used for controlling the lighting device <NUM>. The lighting control unit <NUM> generates the lighting control signal by converting the selected lighting signal into the control protocol of the lighting device <NUM>. The lighting control unit <NUM> outputs the generated lighting control signal to the lighting device <NUM>.

In the above description, a case where the lighting control unit <NUM> controls the lighting device <NUM> using the input information acquired from the generation unit <NUM> has been described as an example. The lighting control unit <NUM> may control the lighting device <NUM> based on the state information <NUM> as described below instead of the input information.

The lighting control unit <NUM> acquires the state information <NUM> as the information for controlling the lighting device <NUM>. In this case, the lighting control unit <NUM> generates a lighting control signal for emitting light of a color corresponding to the state information <NUM> using, for example, a color table in which the state information <NUM> and the color are associated with each other, and outputs the generated lighting control signal to the lighting device <NUM>. The color table is stored in, for example, the connection information <NUM>, and the lighting control unit <NUM> acquires the color table with reference to the connection information <NUM>.

For example, a case where the state information <NUM> is a value corresponding to the representative color in the region obtained by dividing the target image into the plurality of regions is considered. In this case, for example, the lighting control unit <NUM> controls the lighting device <NUM> arranged on a right side using the state information <NUM> corresponding to the representative color in the region on a right side of the target image. The lighting control unit <NUM> controls the lighting device <NUM> arranged on a left side using the state information <NUM> corresponding to the representative color in the region on a left side of the target image. Thereby, light of a color the same as that on the right side (or the left side) in the virtual space is emitted on the right side (or the left side) in the real space, and it is possible to create an effect as if the virtual space and the real space are seamlessly spread. It is assumed that the virtual space in this case is a virtual space displayed on a monitor provided on a wall surface of a room or the like. That is, it is assumed that the virtual space is not a virtual space displayed on the HMD or the like.

The display control unit <NUM> generates a display control signal and outputs the generated signal to the display device <NUM> to control an image to be displayed on the display device <NUM>. The display control unit <NUM> generates, for example, three-dimensional information of a virtual space including an image corresponding to an image signal in distribution information, for example, a virtual space indicating a live venue in which a live video is displayed on a screen, and generates the generated three-dimensional information as the display control signal. The display control unit <NUM> outputs the generated display control signal to the display device <NUM>. For example, the display control unit <NUM> acquires the connection information <NUM> with reference to the storage unit <NUM>. The display control unit <NUM> generates the three-dimensional information in the virtual space based on information such as a size of the monitor in the display device <NUM> indicated by the acquired connection information <NUM>. For example, when the size of the monitor in the display device <NUM> is sufficiently large, the display control unit <NUM> generates three-dimensional information for displaying a virtual space in which a region of spectator seats is set to be wide, and creates a three-dimensional effect of the live venue P1. On the other hand, when the size of the monitor in the display device <NUM> is small, three-dimensional information for displaying a virtual space in which a main stage is the main (a region of spectator seats is narrow) is generated, and an effect that a performance on the stage is easily viewed is created. The three-dimensional data (for example, the three-dimensional data indicating a position of the stage or a position of the lighting in the virtual space) (hereinafter, referred to as virtual space information) necessary for generating the virtual space is acquired as information different from the connection information <NUM> of the monitor. For example, the virtual space information may be distributed by the distribution server <NUM>, the virtual space information may be embedded in a sound signal or an image signal, or the virtual space information may be acquired in advance by being embedded in a program.

The connection information <NUM> may be acquired by any method. For example, the distribution server <NUM> may distribute the connection information <NUM> corresponding to a viewing environment of the remote location P2, the connection information <NUM> may be embedded in a program in advance, or may be embedded in an image signal, a sound signal or the like and distributed.

When the display control unit <NUM> acquires an input signal indicating a movement of an actual line of sight or the like of the user, the display control unit <NUM> specifies a viewing position of the user in the virtual space based on the input information. The display control unit <NUM> generates, based on the specified viewing position in the virtual space, three-dimensional information of the virtual space in which the live venue viewed from the viewing position is indicated, and generates the generated three-dimensional information as a display control signal. The display control unit <NUM> outputs the generated display control signal to the display device <NUM>.

The sound control unit <NUM> generates a sound control signal and outputs the generated signal to the speaker device <NUM>, thereby controlling sound to be output from the speaker device <NUM>. For example, the sound control unit <NUM> generates a lighting control signal by converting a sound signal in the distribution information into a control protocol of lighting device <NUM>.

The sound control unit <NUM> acquires a viewing position of the user in the virtual space specified by the display control unit <NUM> from, for example, the display control unit <NUM>. Based on the acquired viewing position in the virtual space, the sound control unit <NUM> generates sound of the live venue heard from the viewing position.

The device control unit <NUM> integrally controls the control device <NUM>. For example, the device control unit <NUM> synchronizes time codes of a lighting signal, an image signal and a sound signal based on the distribution information distributed from the distribution server <NUM>. As a method of synchronizing the time codes, any method of synchronizing the time codes, which has been performed in related-art, may be adopted. For example, the signals are synchronized using time codes embedded in the respective signals including the lighting signal, the image signal and the sound signal. The device control unit <NUM> outputs the synchronized lighting signal, image signal, and sound signal to the generation unit <NUM>, the lighting control unit <NUM>, the display control unit <NUM> and the sound control unit <NUM>, respectively.

<FIG> is a block diagram showing an example of a configuration of the lighting device <NUM> according to the embodiment. The lighting device <NUM> includes, for example, a control signal acquisition unit <NUM> and a light emitting unit <NUM>. The control signal acquisition unit <NUM> acquires a lighting control signal from the control device <NUM> and outputs the acquired signal to the light emitting unit <NUM>. The light emitting unit <NUM> emits light according to the lighting control signal from the control signal acquisition unit <NUM>.

<FIG> is a block diagram showing an example of a configuration of the display device <NUM> according to the embodiment. The display device <NUM> includes, for example, a control signal acquisition unit <NUM> and a display unit <NUM>. The control signal acquisition unit <NUM> acquires a display control signal from the control device <NUM> and outputs the acquired signal to the display unit <NUM>. The display unit <NUM> displays an image corresponding to the display control signal from the control signal acquisition unit <NUM>.

<FIG> is a block diagram showing an example of a configuration of the speaker device <NUM> according to the embodiment. The speaker device <NUM> includes, for example, a control signal acquisition unit <NUM> and a sound output unit <NUM>. The control signal acquisition unit <NUM> acquires a sound control signal from the control device <NUM> and outputs the acquired signal to the sound output unit <NUM>. The sound output unit <NUM> outputs sound corresponding to the sound control signal from the control signal acquisition unit <NUM>.

<FIG> is a block diagram showing an example of a configuration of the input device <NUM> according to the embodiment. The input device <NUM> includes, for example, an input information acquisition unit <NUM> and an input information output unit <NUM>. Information indicating a movement of an actual line of sight or the like of the user is input to the input information acquisition unit <NUM>. The input information output unit <NUM> outputs input information input to the input information acquisition unit <NUM> to the control device <NUM>.

In the above description, a case where the input device <NUM> is the HMD and is a device that detects an actual line of sight, a movement of a head or the like of the user has been described as an example, but the present invention is not limited thereto. The input device <NUM> may be an input device such as a mouse, a keyboard, a game controller or a touch pad. In this case, when the user operates the mouse or the like, a line-of-sight direction or the like in the virtual space is operated, and the operated information is input to the input information acquisition unit <NUM>. The input information output unit <NUM> outputs input information input to the input information acquisition unit <NUM> to the control device <NUM>.

<FIG> is a flowchart showing a flow of processing executed by the control device <NUM> according to the embodiment. The control device <NUM> acquires a viewing position of the user in the virtual space (step S10). The control device <NUM> determines the viewing position of the user in the virtual space according to an input signal from the input device <NUM>. The control device <NUM> acquires distribution information (step S11). The control device <NUM> generates a display control signal based on an image signal in the distribution information (step S12). In this case, the control device <NUM> generates the display control signal such that an image corresponding to the viewing position is displayed on the display device <NUM>. The control device <NUM> generates three-dimensional information of a virtual space simulating a live venue in which an image (live video) corresponding to the image signal is projected on a screen, and sets the generated three-dimensional information as the display control signal. The control device <NUM> outputs the generated display control signal to the display device <NUM> (step S13).

The control device <NUM> generates a sound control signal based on a sound signal in the distribution information (step S14). In this case, the control device <NUM> generates the sound control signal such that sound corresponding to the viewing position is output from the speaker device <NUM>. The control device <NUM> outputs the generated sound control signal to the speaker device <NUM> (step S15). In this case, the control device <NUM> generates, for example, the sound control signal corresponding to the viewing position in the virtual space, and controls how to hear the sound according to the viewing position in the virtual space.

The control device <NUM> generates a lighting control signal based on the viewing position of the user in the virtual space (step S16). The control device <NUM> outputs the generated lighting control signal to the lighting device <NUM> (step S17). The control device <NUM> returns to the processing shown in step S10.

In the flowchart described above, sound output processing of outputting the sound of the speaker device <NUM> is executed after display processing of displaying the image on the display device <NUM>, and then light emission processing of causing the lighting device <NUM> to emit light is executed. However, the present invention is not limited thereto. The display processing, the sound output processing and the light emission processing may be executed in any order. The display processing, the sound output processing and the light emission processing may be executed in parallel.

In the flowchart described above, the control device <NUM> may execute light emission processing (hereinafter, referred to as second light emission processing) based on the state information <NUM>, in which light to be emitted by the lighting device <NUM> is controlled according to the state information <NUM>. The second light emission processing may be executed in any order in the same manner as the display processing, the sound output processing and the light emission processing, or each processing may be executed in parallel.

As described above, the image display system <NUM> according to the embodiment includes the first acquisition unit <NUM>, the display control unit <NUM> and the lighting control unit <NUM>. The first acquisition unit <NUM> acquires image information. The image information is information indicating the image (target image) in which the situation of the live performance (viewing target) is captured. The first acquisition unit <NUM> is an example of an "image information acquisition unit". The display control unit <NUM> acquires position information. The position information is a position where the user views the target image in a virtual space. The virtual space is a virtual space in which the image (target image) in which the situation of the live performance (viewing target) is captured is displayed. The display control unit <NUM> is an example of a "position information acquisition unit". The lighting control unit <NUM> controls light to be emitted by the lighting device <NUM> based on the position information. The lighting control unit <NUM> is an example of a "light emission control unit".

Thereby, the image display system <NUM> according to the embodiment can provide the user with an experience as if the user walks around the virtual space simulating a live venue. Alight emission state of the lighting device <NUM> can be changed according to the viewing position in the virtual space. Therefore, the user can feel presence of the live performance more strongly.

In the image display system <NUM> according to the embodiment, the lighting control unit <NUM> controls light emission of the lighting device <NUM> based on the state information <NUM>. The state information <NUM> is information indicating a state of a live space (target space) in which the live performance (viewing target) is present. Thereby, the image display system <NUM> according to the embodiment can change the light emission state of the lighting device <NUM> according to a state of an effect by sound or light in the live venue. Therefore, the user can feel a sense of unity with the live performance.

The image display system <NUM> according to the embodiment further includes a sound control unit <NUM>. The first acquisition unit <NUM> acquires sound information. The sound information is information indicating sound collected in the live space (target space) in which the live performance (viewing target) is present. The first acquisition unit <NUM> is an example of a "sound information acquisition unit". The sound control unit <NUM> controls the sound to be output from the speaker device <NUM> based on the sound information. The sound control unit <NUM> is an example of a "speaker control unit". Thereby, the image display system <NUM> according to the embodiment can cause the speaker device <NUM> to output the sound in the live venue. Therefore, an effect the same as the effect described above can be obtained, and the user can feel presence of the live performance more strongly. The sound output from the speaker device <NUM> may be changed according to the viewing position of the user in the virtual space. In this case, for example, the sound output from the speaker device <NUM> is moved or changed according to the viewing position by using a virtual effect such as a head-related transfer function (HTRF).

Here, a first modification of the embodiment will be described. The present modification is different from the above-described embodiment in that a virtual lighting device (hereinafter, referred to as virtual lighting) is set in a virtual space.

When generating three-dimensional information corresponding to the virtual space, the display control unit <NUM> sets virtual lighting in the virtual space. For example, the display control unit <NUM> sets the virtual lighting as a light source in the virtual space. The virtual lighting is set as the light source of the virtual space, for example, according to a configuration of stage lighting in a live venue. Although it is not necessary for the virtual lighting to completely match the configuration of the stage lighting in the live venue, it is desirable that the virtual lighting has a configuration the same as the configuration of the stage lighting in the live venue as much as possible and can create a lighting effect the same as that in the live venue.

The display control unit <NUM> controls light to be emitted by the virtual lighting. For example, the display control unit <NUM> causes the virtual lighting to emit the light according to a lighting signal of the specific stage lighting corresponding to the virtual lighting in distribution information.

The display control unit <NUM> specifies a viewing position and a line-of-sight direction of the user in the virtual space based on input information, and changes an appearance of the entire virtual space based on the specified viewing position and line-of-sight direction. Lighting in the virtual space can be selected according to a predetermined correspondence rule. As a result, the appearance of the entire virtual space viewed by the viewer changes. The display control unit <NUM> causes the virtual lighting to emit the light according to the lighting signal of the selected stage lighting. In this case, the display control unit <NUM> acquires information indicating an arrangement position of the virtual lighting from, for example, the input device <NUM>. Alternatively, the display control unit <NUM> may acquire the information indicating the arrangement position of the virtual lighting from information indicated in the connection information <NUM>. In this case, in the connection information <NUM>, an item related to the virtual lighting is provided, and information indicating an installation position of the virtual lighting is stored in the item.

For example, the display control unit <NUM> arranges the virtual lighting at a position the same as a position where the stage lighting of the live venue P1 is installed in the virtual space. The display control unit <NUM> associates the virtual lighting arranged in the virtual space with the stage lighting of the live venue P1, and controls light emission of the virtual lighting according to light emission of the stage lighting. Alternatively, the display control unit <NUM> may associate the stage lighting of the live venue P1 with the virtual lighting arranged in the virtual space in advance, and control the light emission of the virtual lighting according to the light emission of the stage lighting based on the association. In this case, when the virtual lighting arranged in the virtual space is arranged in the same manner as the stage lighting in the live venue P1, the virtual lighting and the stage lighting may be automatically associated with each other.

The display control unit <NUM> causes the virtual lighting to emit light of a color corresponding to the state information <NUM> by using, for example, a color table in which the state information <NUM> and the color are associated with each other.

<FIG> is a flowchart showing a flow of processing performed by the control device <NUM> according to the modification of the embodiment. The control device <NUM> acquires distribution information (step S20). The control device <NUM> generates three-dimensional information of a virtual space in which virtual lighting is set in a virtual live venue, based on an image signal in the distribution information (step S21). The generated three-dimensional information is output to the display device <NUM> as a display control signal (step S22).

The control device <NUM> generates the state information <NUM> based on the distribution information (step S22). The control device <NUM> generates, for example, information for controlling stage lighting provided in a live venue, such as a DMX signal as the state information <NUM>. The control device <NUM> generates, for example, information indicating a representative color representing a color of an image (target image) in which a situation of a live performance is captured, as the state information <NUM>. the control device <NUM> generates, for example, information indicating a color corresponding to a frequency of sound collected in a live space, as the state information <NUM>. The control device <NUM> controls light emission of the virtual lighting based on the generated state information <NUM> (step S23).

The control device <NUM> acquires a viewing position of the user in the virtual space (step S24). The control device <NUM> controls the light emission of the virtual lighting according to the acquired viewing position (step S25). The control device <NUM> generates a display control signal according to the viewing position such that an image viewed from the viewing position is displayed on the display device <NUM> (step S26). The control device <NUM> outputs the generated display control signal to the display device <NUM> (step S27). The control device <NUM> generates a sound control signal according to the viewing position (step S28). The control device <NUM> outputs the generated sound control signal to the speaker device <NUM> (step S29). The control device <NUM> returns to the processing shown in step S20.

In the flowchart described above, as in <FIG>, display processing, sound output processing, light emission processing (processing of causing the virtual lighting to emit light in <FIG>), processing of controlling the light to be emitted by the virtual lighting according to the viewing position in the virtual space, processing of controlling the light to be displayed on the display device <NUM> according to the viewing position in the virtual space, and processing of controlling the sound to be output from the speaker device <NUM> according to the viewing position in the virtual space may be executed in any order. The above-described processing may be executed in parallel. Both the virtual lighting and the display device <NUM> (real lighting) may be controlled at the same time.

As described above, the image display system <NUM> according to the modification of the embodiment includes the first acquisition unit <NUM> and the display control unit <NUM>. The first acquisition unit <NUM> acquires image information. The image information is information indicating the image (target image) in which the situation of the live performance (viewing target) is captured. The first acquisition unit <NUM> is an example of an "image information acquisition unit". The display control unit <NUM> displays the virtual space. The virtual space is a virtual space in which the image (target image) in which the situation of the live performance (viewing target) is captured is displayed. The display control unit <NUM> controls the light to be emitted by the virtual lighting (virtual lighting device) provided in the virtual space based on the state information <NUM>. The state information <NUM> is, for example, information indicating a state of sound or light in the live space. The state information <NUM> is, for example, information for controlling the stage lighting provided in the live venue, such as the DMX signal. The state information <NUM> is information indicating the representative color representing the color of the image (target image) in which the situation of the live performance is captured. The state information <NUM> is information indicating the color corresponding to the frequency of the sound collected in the live space. Thereby, the image display system <NUM> according to the embodiment can change a light emission state of the virtual lighting according to a state of an effect by sound or light in the live venue. Therefore, an effect the same as the effect described above can be obtained, and the user can feel a sense of unity with the live performance.

In the above-described embodiment, a case where the DMX signal for controlling the stage lighting is distributed as the lighting signal from the distribution server <NUM> has been described as an example, but the present invention is not limited thereto. In general, in a lighting device, a command sequence of the DMX signal is different depending on the device. As a countermeasure, for example, the lighting device <NUM> (referred to as a standard device) of the user may be registered in the distribution server <NUM>, and the distribution server <NUM> may distribute a signal obtained by converting the DMX signal of the stage lighting into the DMX signal of the standard device as the lighting signal. Processing of converting into the DMX signal of the standard device may be performed by the control device <NUM> instead of the distribution server <NUM>.

The following modes may be considered as handling of the lighting signal included in the distribution information.

In the mode (<NUM>), a timing of the image signal and a timing of the lighting signal may be synchronized with each other by synchronizing the lighting signal with the image signal and the sound signal using a time code or the like, and the synchronized signals may be used. In the mode (<NUM>), it is not necessary to perform synchronization processing, and distribution information may be used directly, but when the lighting signal is to be corrected later, it is necessary to recreate a sound signal (or image signal). For example, the distribution server <NUM> may generate the distribution information in which the lighting signal corresponding to a lighting environment is embedded in the sound signal (or image signal) according to the environment of the lighting device <NUM> of the user. In the mode (<NUM>), a lighting signal corresponding to the embedded lighting signal identification ID is included in distribution information and distributed, or a lighting signal corresponding to the embedded URL is downloaded, so that the synchronized lighting signal can be used. According to the mode (<NUM>), even when the lighting signal is to be corrected later, it is not necessary to recreate a sound signal (or image signal). By applying the mode (<NUM>), not only the lighting signal but also meta information (for example, information of a live player) may be distributed in association with the lighting signal identification ID and the URL. In this case, the generation unit <NUM> may generate a lighting control signal or the state information <NUM> using the lighting signal included in each of the modes (<NUM>) to (<NUM>). In particular, when the lighting signal is embedded in the sound signal (or image signal) in the mode (<NUM>), for example, the generation unit <NUM> extracts the lighting signal embedded in the sound signal (or image signal) and generates a lighting control signal based on the extracted lighting signal.

In the above-described embodiment, a case where the lighting device <NUM> is controlled, and in the modification example thereof, a case where the virtual lighting is controlled have been described as examples, but the present invention is not limited thereto. In the image display system <NUM>, both the lighting device <NUM> and the virtual lighting may be controlled. Both the lighting device <NUM> and the virtual lighting may be set to an ON state or an OFF state by an operation of the user.

When there are a plurality of speaker devices <NUM>, sound output from each of the plurality of speaker devices <NUM> may be changed to express a sound field three-dimensionally. The sound field may be changed according to a viewpoint position of the user in the virtual space. When the speaker device <NUM> is a headphone, a three-dimensional sound field may be output by performing 3D sound processing using the headphone.

Controlling the light to be emitted by the lighting device <NUM> or the virtual lighting includes controlling an intensity, a color, a direction and an emission mode of the light to be emitted. The light emission mode is a mode of emitting light, and includes, for example, a mode of lighting, blinking, gradually brightening, gradually darkening and the like.

In the above-described embodiment, a case where the situation of the live performance is reproduced in real time has been described as an example, but the present invention is not limited thereto. The present invention can also be applied to a case where a video or the like in which a situation of a live performance is recorded is reproduced at a home of the user or the like.

In the above-described embodiment, a case where the lighting signal is the DMX signal for controlling the stage lighting has been described as an example, but the present invention is not limited thereto. The lighting signal may be a signal indicating illuminance or the like measured in the live venue P1.

The HMD may be used as the display device <NUM>. In this case, the entire live venue in which the stage, the spectator seats, the virtual lighting and the like are provided is displayed on a screen of the HMD as the virtual space. Then, the control device <NUM> performs control according to the viewing position in the virtual space such that the entire virtual space moves integrally. That is, the control device <NUM> does not perform control such that only the virtual lighting moves alone while the virtual space is fixed. On the other hand, in a case of the physical display device <NUM> (real lighting) as described in the above-described embodiment, the display device <NUM> is fixedly arranged in the real world. Therefore, even when the virtual space moves, a position of the real lighting cannot be move. For example, when the virtual space moves and the real lighting is installed near a position where an image of the virtual lighting is displayed, the control device <NUM> performs control such that the real lighting located nearby and the virtual lighting move in conjunction with each other. Thereby, it is possible to create an effect as if the real lighting is moving in conjunction with the viewing position in the virtual space.

All or a part of the image display system <NUM> and the control device <NUM> according to the above-described embodiment may be implemented by a computer. In this case, a program for implementing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by a computer system. Here, the "computer system" includes an OS and hardware such as peripheral devices. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM or a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when the program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory in a computer system serving as a server or a client in that case. In addition, the program may be a program for implementing a part of the above-described function, may be a program capable of realizing the above-described function in combination with a program already recorded in the computer system, or may be a program realized using a programmable logic device such as an FPGA.

Claim 1:
A computer-implemented display control method comprising:
acquiring, by an image information acquisition unit (<NUM>), image information, the image information indicating a target image in which a situation of a live performance being performed in a live venue (P1) is captured; and
displaying, by a display device (<NUM>), a virtual space in which the target image corresponding to the image information is displayed;
and
setting, by a display control unit (<NUM>), a virtual lighting as a light source in the virtual space;
by the display control unit (<NUM>), generating a display control signal and outputting the generated display control signal to control the virtual space, to be displayed by the display device (<NUM>); characterized by
controlling, by the display control unit (<NUM>), light to be emitted by the virtual lighting based on state information (<NUM>) indicating a state of a target space in the target image, wherein the target space is a live space in which the live performance is present.