IMAGE SHARING METHOD AND IMAGE SHARING SYSTEM, AND STORAGE MEDIUM

An image sharing method including generating image data of an image from a viewpoint at a position based on image data obtained from a plurality of image capturing apparatuses that capture images of the same subject simultaneously, setting a cyber space for viewing the image data from the viewing apparatuses, and distributing the image data to each of the plurality of viewing apparatuses. Information on an avatar, an initial viewing position in the cyber space, and the motion information of a viewer of each viewing apparatus is obtained and environment data pertaining to an avatar of another viewing apparatus based on a viewpoint in the cyber space obtained based on the initial viewing position and the motion information is sent to each viewing apparatus.

BACKGROUND

Field of the Disclosure

The present disclosure relates to an image sharing method and an image sharing system, and a storage medium.

Description of the Related Art

Recent years have seen an increase in opportunities to stream music festivals, live performances, theater productions, sports, and the like. It is also becoming more common to see environments where free viewpoint images, which can be seen from any desired viewpoint, are generated based on multi-viewpoint image data obtained by using a plurality of image capturing apparatuses placed at different locations to shoot images simultaneously. In the future, such multi-viewpoint image data is expected to be distributed as packaged content over broadcast waves, or via disks or subscription video on demand, in addition to conventional single-viewpoint images shot from a single viewpoint.

Physical interplay between performers, participants, and other attendees is one of the unique pleasures of attending an event such as a live performance or a sports game in person at a real venue. For example, many things can happen at a live music performance, such as a performer diving into the front row of the crowd or crowd surfing, attendees lifting each other up, and call and response between performers and attendees, as well as attendees making contact with each other, losing sight of the stage, and more. However, such physical interplay is absent in conventional one-way online video sharing, making it difficult for viewers to feel the same sense of immersion and participation as if they were actually at the venue.

Japanese Patent Laid-Open No. 08-336166 discloses a video viewing apparatus capable of enabling a plurality of viewers to view the same space. By sending information on the position and direction of the viewpoint of each viewer in a cyber space from remote terminals to a host terminal, the video viewing apparatus enables a plurality of viewers to view images and audio from their respective viewpoints.

In this manner, the plurality of viewers supply information pertaining to their respective viewpoints and audio to the distributor of the multi-viewpoint image data, which makes it possible for any number of viewers to view the distributed content while sharing the same cyber space.

However, although the technique disclosed in Japanese Patent Laid-Open No. 08-336166 allows any number of viewers to view video from their respective viewpoints in the same cyber space, there is no interplay between viewers, the subjects in the video distributed, and so on in the video.

SUMMARY

The present disclosure has been made in consideration of making it possible for viewers to feel a heightened sense of immersion when viewing distributed content.

According to the present disclosure, provided is an image sharing method including generating first image data of an image from a viewpoint at a position based on image data obtained from a plurality of image capturing apparatuses that capture images of the same subject simultaneously; setting a cyber space for viewing the first image data from a plurality of viewing apparatuses; distributing the first image data to each of the plurality of viewing apparatuses; obtaining, from each viewing apparatus among the plurality of viewing apparatuses, information on an avatar, an initial viewing position in the cyber space, and motion information of a viewer who uses the viewing apparatus for the viewing; and sending, to each viewing apparatus, environment data pertaining to an avatar of another viewing apparatus that enters a field of view of the viewing apparatus in the cyber space, based on a viewpoint in the cyber space obtained based on the initial viewing position and the motion information.

Further, according to the present disclosure, provided is an image sharing system comprising one or more processors and/or circuitry which function as: a distributing unit that generates first image data capable of generating an image from a viewpoint at a position based on image data obtained from a plurality of image capturing apparatuses that capture images of the same subject simultaneously, and distributes the first image data to a plurality of viewing apparatuses; a setting unit that sets a cyber space for viewing the first image data from the plurality of viewing apparatuses, prior to the distribution; an obtainment unit that obtains, from each viewing apparatus among the plurality of viewing apparatuses, information on an avatar, an initial viewing position in the cyber space, and motion information of a viewer who uses the viewing apparatus for the viewing; and a processing unit that sends, to each viewing apparatus, environment data pertaining to an avatar of another viewing apparatus that enters a field of view of the viewing apparatus in the cyber space, based on a viewpoint in the cyber space obtained based on the initial viewing position and the motion information.

Furthermore, according to the present disclosure, provided is a storage medium storing a program that is executable by the computer, wherein the program includes program code for causing the computer to perform an image sharing method comprising: generating first image data of an image from a viewpoint at a position based on image data obtained from a plurality of image capturing apparatuses that capture images of the same subject simultaneously; setting a cyber space for viewing the first image data from a plurality of viewing apparatuses, prior to the distributing; distributing the first image data to each of a plurality of viewing apparatuses; obtaining, from each viewing apparatus among the plurality of viewing apparatuses, information on an avatar, an initial viewing position in the cyber space, and motion information of a viewer who uses the viewing apparatus for the viewing; and sending, to each viewing apparatus, environment data pertaining to an avatar of another viewing apparatus that enters a field of view of the viewing apparatus in the cyber space, based on a viewpoint in the cyber space obtained based on the initial viewing position and the motion information.

DESCRIPTION OF THE EMBODIMENTS

FIG.1is a conceptual diagram illustrating the configuration of a video distribution system according to the present embodiment.

The video distribution system according to the present embodiment includes, on a distribution side, a plurality of camera apparatuses100, a main server apparatus200, storage250connected to the main server apparatus200, a sub server apparatus300, and storage350connected to the sub server apparatus300. The system also includes a plurality of viewing apparatuses400on a viewing side, and the plurality of viewing apparatuses400, the main server apparatus200, and the sub server apparatus300are connected over a network500.

The plurality of camera apparatuses100are connected to the main server apparatus200, and each has a function for shooting an image and a function for collecting audio. As will be described later, images are shot and sound is collected simultaneously from a plurality of predetermined positions relative to a subject to be shot, and the obtained image data and audio data are output to the main server apparatus200.

The configurations of the main server apparatus200, the sub server apparatus300, and the viewing apparatus400, which constitute the video distribution system, will be described further hereinafter.

FIG.2is a block diagram illustrating the configuration of the main server apparatus200.

In the main server apparatus200, an input unit203, a communication unit204, an image processing unit205, a recording/playback unit206, a display unit207, an audio processing unit208, and an operation unit209are connected to a CPU201over a CPU bus210, and are controlled by the CPU201. The input unit203, the communication unit204, the image processing unit205, the recording/playback unit206, the display unit207, the audio processing unit208, and the operation unit209are also connected to a work memory211over a memory bus212.

The CPU201is a processor that controls the elements of the main server apparatus200. A primary storage unit202is constituted by a ROM, a RAM, or the like, and stores processing programs, data, and the like for causing the CPU201to operate, including performing processing for distributing content.

The plurality of camera apparatuses100are connected over the input unit203, and input viewpoint information such as the positions, orientations, and the like of the plurality of camera apparatuses100, as well as image data and audio data from the camera apparatuses100. The embodiments of the present disclosure assume that a large amount of data captured by the plurality of cameras is processed quickly and securely at the same time, and the elements of the system are directly connected through a dedicated interface or the like. However, if synchronized control, data transfer, and information security can be ensured among the cameras, the plurality of camera apparatuses100may be connected via the communication unit204over the network500.

Based on the image data obtained by the plurality of camera apparatuses100capturing images of the same subject simultaneously, and the viewpoint information from each camera apparatus100, the image processing unit205reconfigures the data into three-dimensional data of the subject present in an image space (called “3D spatial data” hereinafter). This 3D spatial data can be used to generate a two-dimensional image viewed from a desired viewpoint, or image data of an image in a cyber space. In the following descriptions, the term “viewpoint” used alone is assumed to refer to both the position and the orientation of a viewpoint, whereas the phrases “position of a viewpoint” and “orientation of a viewpoint” are assumed to refer specifically to the position and the orientation, respectively.

The audio processing unit208performs predetermined processing on the audio data input from the camera apparatuses100. The CPU201then collects the 3D spatial data and audio data in a predetermined format for each frame, and generates 3D spatial data with audio.

The generated 3D spatial data with audio is stored in the mass storage250, an external storage medium such as a disk-based memory or a card-based memory (not shown), or the like, through the recording/playback unit206. The 3D spatial data with audio stored in the mass storage250is read out through the recording/playback unit206as necessary, and can be distributed to each viewing apparatus400as content from the communication unit204over the network500.

The communication unit204is used to connect to the network500, either wirelessly or over wires.

The display unit207displays images input from the input unit203, images, text, icons, and the like for controlling the main server apparatus200, and the like. A user can control the main server apparatus200by operating an operation member connected to the operation unit209based on the displayed content. For example, by executing an editing application for images, audio, and the like stored in the primary storage unit202, the CPU201displays information on the image data, audio data, and the like in the display unit207, and the user can edit the 3D spatial data with audio by operating the operation member.

The work memory211is used to temporarily hold various types of data that have been input, data during processing performed by the image processing unit205and the audio processing unit208, variables, coefficients, and the like necessary for processing, and the like.

The operation unit209detects an operation made by the user on an operation member such as a remote controller, a button, or a touch panel (not shown), and communicates operation information based on the detected operation to the CPU201. The CPU201performs control based on the communicated operation information.

The configuration of the sub server apparatus300will be described next with reference toFIG.3.

In the sub server apparatus300, a communication unit303, an image processing unit304, a recording/playback unit305, a display unit306, an operation unit307, an audio processing unit308, and a tactile processing unit312are connected to a CPU301over a CPU bus311, and are controlled by the CPU301. The communication unit303, the image processing unit304, the recording/playback unit305, the display unit306, the operation unit307, the audio processing unit308, and the tactile processing unit312are also connected to a work memory309over a memory bus310.

The CPU301is a processor that controls the elements of the sub server apparatus300. A primary storage unit302is constituted by a ROM, a RAM, or the like, and stores processing programs, data, and the like for causing the CPU301to operate, including performing processing for generating environment image data, environment audio data, and haptics data (sense information data), which will be described later.

The communication unit303is used to connect to the network500, either wirelessly or over wires. The communication unit303obtains information pertaining to a virtual audience seat space (described later), from the main server apparatus200over the network500, and inputs information such as an initial position, avatar information, device configuration information, audio data, motion information, and the like of each viewing apparatus400, from each viewing apparatus400. If a haptics device is included, the device configuration information includes body part information indicating the type of the haptics device and the part of the body to which the haptics device is attached is included.

The image processing unit304and the audio processing unit308combine images of avatars of the plurality of viewing apparatuses400with audio based on information pertaining to the virtual audience seat space input from the communication unit303, as well as the avatar information, initial position, audio data, motion information, and the like of the plurality of viewing apparatuses400. Hereinafter, data of the image of an avatar obtained through such compositing will be referred to as “environment image data”, and data of the audio obtained through such compositing will be referred to as “environment audio data”.

The tactile processing unit312determines, based on the device configuration information, whether the viewing apparatus400includes a haptics device414(described later), and if so, generates the haptics data.

Although the present embodiment assumes that the haptics data generates a sense of hardness, roughness, friction, temperature, and the like from the outline shape, color, and texture of the avatar image, the method is not particularly limited thereto as long as the same effect can be achieved. For example, the information of the avatar may include information pertaining to hardness, roughness, friction, temperature, and the like.

Alternatively, instead of the haptics data corresponding to the overall image of the avatar, the device configuration information may be referenced and the haptics data generated only for regions where tactile feedback can be made, according to the number of haptics devices and in the corresponding viewing apparatus400and the parts of the body where the devices are worn.

The environment image data, the environment audio data, and the haptics data (when present) will be collectively referred to as “environment data”. The obtained environment data can be sent from the communication unit303to the viewing apparatus400over the network500.

The display unit306can display images, text, icons, and the like for controlling the sub server apparatus300, and the user can control the sub server apparatus300by operating an operation member connected to the operation unit307based on the displayed content. For example, by causing the CPU201to execute an application stored in the primary storage unit302, displaying information on the virtual audience seat space in the display unit306, and operating the operation member, the arrangement of seats, the permissible level of congestion, and the like can be set, for example.

The work memory309is used to temporarily hold various types of data that have been input, data during processing performed by the image processing unit304, the audio processing unit308, and the tactile processing unit312, variables, coefficients, and the like necessary for processing, and the like.

The operation unit307detects an operation made by the user on an operation member such as a remote controller, a button, or a touch panel (not shown), and communicates operation information based on the detected operation to the CPU301. The CPU301performs control based on the communicated operation information.

The configuration of the viewing apparatus400will be described next with reference toFIG.4.

In the viewing apparatus400, a communication unit403, an image processing unit404, a display unit405, an audio processing unit406, an operation unit407, a microphone408, a speaker409, a motion detection unit410, and the haptics device414are connected to a CPU401over a CPU bus412, and are controlled by the CPU401. The communication unit403, the image processing unit404, the display unit405, the audio processing unit406, the operation unit407, the microphone408, the speaker409, the motion detection unit410, and the haptics device414are also connected to a work memory413over a memory bus411.

The CPU401is a processor that controls the elements of the viewing apparatus400. A primary storage unit402is constituted by a ROM, a RAM, or the like, and stores processing programs, data, and the like for causing the CPU401to operate, including viewing processing (described later).

The communication unit403is used to connect to the network500, either wirelessly or over wires. The communication unit403receives, over the network500, content distributed from the main server apparatus200, as well as the virtual seat information and environment data sent from the sub server apparatus300. An initial position of the viewpoint of the viewing apparatus400(an initial viewing position) in the cyber space of content distributed from the main server apparatus200is set by using the operation unit407to select an unselected audience seat among the audience seats indicated by the virtual audience seat information. The communication unit403outputs information, such as the set initial position of the viewpoint of the viewing apparatus400, avatar information, device configuration information, audio data, motion information, and the like, from the communication unit403to the sub server apparatus300over the network500.

The image processing unit404generates image data of an image as seen from the viewpoint of the viewing apparatus400, from the 3D spatial data of the content received through the communication unit403, composites the image data with the environment image data, and converts the composited image data into a format compliant with the display unit405. The viewpoint of the viewing apparatus400can be found based on the initial position of the viewpoint of the viewing apparatus400and a history of the motion information.

The audio processing unit406composites the audio data of the content received by the communication unit403over the network500with the environment audio data, and converts the composited audio data into a format compliant with the speaker409. The audio processing unit406also converts the audio input from the microphone408into audio data, and the audio data resulting from the conversion is sent from the communication unit403to the sub server apparatus300over the network500.

The haptics device414provides tactile feedback to the user by operating based on the haptics data included in the environment data.

The display unit405displays the image data processed by the image processing unit205, displays images, text, icons, and the like for controlling the viewing apparatus400, and provides displays for generating avatars. The user can control the viewing apparatus400by operating an operation member connected to the operation unit407based on the displayed content.

The work memory413is used to temporarily hold various types of data that have been input, data during processing performed by the image processing unit404and the audio processing unit406, variables, coefficients, and the like necessary for processing, and the like.

The operation unit407detects an operation made by the user on an operation member such as a remote controller, a button, or a touch panel (not shown), and communicates operation information based on the detected operation to the CPU401. The CPU401performs control based on the communicated operation information.

The motion detection unit410converts an amount of change obtained from a motion sensor (not shown) worn by the user, and an amount of change in the user's motion obtained by analyzing an image of the user captured by a camera (not shown) included in the viewing apparatus400, into, for example, 6DoF change amount information. The obtained change amount information is sent to the sub server apparatus300as the motion information mentioned above.

Note that a head-mounted display having a display, a microphone, a speaker, and a motion detection unit may be connected to the viewing apparatus400and used, or the viewing apparatus400itself may be a head-mounted display.

FIG.5is a conceptual diagram illustrating the general flow of data during content distribution in the video distribution system configured as described above. It is assumed that, prior to distributing the content, the sub server apparatus300has already sent, to the main server apparatus200, IP address information of the viewing apparatus400through which the content is to be viewed; the main server apparatus200has already sent configuration information on the cyber space (described later) to the sub server apparatus300; and each viewing apparatus400has already sent the avatar information, the initial position of the viewpoint, and the device configuration information to the sub server apparatus300.

The main server apparatus200distributes content to each of the viewing apparatuses400to which the content is to be distributed.

Each viewing apparatus400also sends the motion information detected by the motion detection unit410and the audio data collected by the microphone408to the sub server apparatus300.

The sub server apparatus300obtains the motion information and audio data (collectively referred to as “reaction information” hereinafter) sent from each viewing apparatus400.

FIG.6Aillustrates an example of the data of the avatar information and the reaction information from each viewing apparatus400, managed by the sub server apparatus300. An avatar ID, the audio data, an action ID, and the 6DoF change amount information of the avatar, sent from each viewing apparatus400, are held and updated as necessary. The change amount information includes right/left, forward/backward, up/down, roll, yaw, and pitch change amounts. If the viewing apparatus400includes the haptics device414, the haptics data is also held and updated as necessary.

The sub server apparatus300aggregates the initial position of the viewpoint, the avatar information, and the reaction information of each viewing apparatus400, generates the environment data for each viewing apparatus400, and sends the environment data to each viewing apparatus400. If the viewing apparatus400includes the haptics device414, the environment data is generated including the haptics data and sent.

Each viewing apparatus400generates an image seen from the current viewpoint from the content distributed from the main server apparatus200, composites the image data and audio data of the generated image with the environment data sent from the sub server apparatus300, and outputs the composite image and the composite audio. If the viewing apparatus400includes the haptics device414, the haptics device414is driven based on the haptics data included in the environment data to provide tactile feedback.

First Embodiment

Content distribution processing performed by the main server apparatus200according to a first embodiment will be described next with reference toFIG.7.

First, in step S101, the main server apparatus200obtains the dimensions of a three-dimensional physical space in which images are shot.FIG.8Aillustrates a plan view of a physical space including a main stage space1including a stage and an audience seat space2including audience seats, as well as an arrangement of the camera apparatuses100, as an example of a physical space in which images are shot. These may be measured by a worker who installed the stage where a performer performs, or may be values provided in the specifications of the venue. A screen monitor3may be disposed, in a position that does not interfere with the shooting by the camera apparatuses100, such that the performer can check the reactions of the viewers. In this case, an image of the environment image data generated from a predetermined viewpoint in the main stage space1is displayed by the sub server apparatus300in the screen monitor3.

Next, in step S102, the positions and orientations of the plurality of camera apparatuses100are obtained.

As the positions of the plurality of camera apparatuses100, coordinates which take the position and orientation of one camera apparatus100among the plurality of camera apparatuses100as a reference can be used to obtain the positions and orientations of the other camera apparatuses100, for example. Note that the reference position and orientation are not limited thereto, and any positions and orientations may be used as long as the relative positions and shooting directions of the plurality of camera apparatuses100can be specified. For example, the positions and orientations of the plurality of camera apparatuses100may be specified by coordinates which take the center of the space to be shot as the origin.

Next, in step S103, the main server apparatus200calculates the configuration of a cyber space corresponding to a three-dimensional physical space in which the images are shot.FIG.8Billustrates a cyber space corresponding toFIG.8A, and illustrates a plan view of a virtual stage space4corresponding to the main stage space1and a virtual audience seat space5corresponding to the audience seat space2. Then, in step S104, the configuration of the cyber space is sent to the sub server apparatus300.

Note that when the main server apparatus200finishes calculating the configuration of the cyber space in step S103, the sub server apparatus300searches for the viewing apparatuses400that wish to view the content before the content distribution is started. Then, in step S105, the IP address information of the viewing apparatuses400that wish to view the content is obtained from the sub server apparatus300.

In step S106, when the plurality of camera apparatuses100start shooting, the main server apparatus200obtains image data and audio data from the plurality of camera apparatuses100, and generates the 3D spatial data with audio (the content). Then, in step S107, distribution destination parameters of the content obtained prior to the distribution are set.

In step S108, the distribution of the content is started for each of the viewing apparatuses400to which the content is to be distributed. At this time, the main server apparatus200adds, to the 3D spatial data with audio, parameters which allow the viewing apparatus400to generate only image data and audio data from the viewpoint of that viewing apparatus400.

The parameters are, for example, key information, generated based on the unique IP address of the viewing apparatus400obtained in the above-described step S105, for decoding 3D spatial data generated through a process including encryption. This prevents unauthorized access from unregistered viewing apparatuses400, and prevents such viewing apparatuses400from surreptitiously viewing the 3D spatial data.

In step S109, whether to end the content distribution is determined. If the distribution is to be ended, the sequence moves to step S110, where the environment data from a predetermined viewpoint in the main stage space1is received from the sub server apparatus300, an image of the environment data is displayed on the screen monitor3, and audio expressed by the environment audio data is streamed using a speaker or the like (not shown). Although the processing of step S110and the screen monitor3are not required, providing these enables the performer to check the reactions of the viewers, which makes it possible to increase the sense of immersion in a physical space.

If it is determined in step S109to end the distribution, the content distribution processing ends.

The generation and output processing of the environment data by the sub server apparatus300according to the present embodiment will be described next with reference toFIG.9.

First, in step S201, the apparatus stands by for the information on the configuration of the cyber space sent from the main server apparatus200in step S104. Once the information on the configuration of the cyber space is received, the sequence moves to step S202, and the virtual audience seat space5is generated.

When the calculation of the configuration of the virtual audience seat space5in step S202ends, in step S203, the main server apparatus200searches for the viewing apparatuses400that wish to view the content before the content distribution is started. Then, the selected audience seats are obtained along with information such as the avatar information, the IP addresses, the device configuration information, and the like of the viewing apparatuses400that wish to view the content.

In addition, in step S204, the IP addresses of the viewing apparatuses400that wish to view the content are sent to the main server apparatus200.

Note that a fee may be charged when an application to view the content is made, and the amount charged may differ depending on the location of the audience seat.

Next, in step S205, it is determined whether or not the main server apparatus200has started distributing the content. If so, the sequence moves to step S206, and if not, the sequence returns to step S203and the processing is repeated.

In step S206, it is determined whether the main server apparatus200has ended the distribution of the content, and if so, the processing ends. However, if the distribution has not ended, the sequence moves to step S207, where the reaction information is obtained from the viewing apparatuses400to which the content is being distributed.

In step S208, it is determined whether the environment data has been generated for each of the viewing apparatuses400to which the content is being distributed. Note that specific examples of the environment data will be described later with reference toFIGS.11A to11DandFIGS.12A to12D.

If a viewing apparatus400for which the environment data has not been generated is present, in step S209, the environment data of that viewing apparatus400is generated, and the sequence returns to step S208. However, if the environment data has been generated for all the viewing apparatuses400to which the content is being distributed, the environment data generated in step S210is output to each viewing apparatus400, and the sequence then returns to step S206. Note that the timing at which the environment data is output is not limited thereto, and the environment data may be output to each corresponding viewing apparatus400in sequence each time the environment data is generated in step S209.

Viewing processing performed by the viewing apparatus400according to the present embodiment will be described next with reference toFIG.10.

First, in step S301, the viewing apparatus400connects to the sub server apparatus300, selects an audience seat in the virtual audience seat space5, applies to view the content, creates an avatar, and sends the created avatar information to the sub server apparatus300. In the example illustrated inFIG.8B, by selecting one of audience seats A to P set in the virtual audience seat space5, each viewing apparatus400can view the content as seen from the position of the selected audience seat.

Next, in step S302, the viewing apparatus400connects to the main server apparatus200, which is the distributor of the content, and then, in step S303, the motion detection unit410and the microphone408obtain the reaction information of the viewer and send the information to the sub server apparatus300over the network500.

In step S304, it is determined whether the main server apparatus200has started distributing the content. If the distribution has not been started, the sequence returns to step S303, and the processing is repeated while standing by for the content distribution to start. When the content distribution is started, the sequence moves to step S305.

In step S305, it is determined whether the main server apparatus200has ended the distribution of the content, and if so, the processing ends. However, if the distribution has not been ended, the content distributed from the main server apparatus200is received in step S306, and the environment data distributed from the sub server apparatus300is then obtained in step S307.

In step S308, image data and audio data as seen from the current viewpoint are generated from the content obtained in step S306and composited with the environment data obtained in step S307, and the composite image data and the composite audio data are then output from the display unit405and the speaker409, respectively. The haptics device414is also driven based on the haptics data included in the environment data.

In step S309, it is determined whether the reaction information of the viewer has been obtained from the motion detection unit410and the microphone408. If the reaction information has not been obtained, the sequence returns to step S305, whereas if the reaction information has been obtained, the sequence moves to step S310, and then returns to step S305after the reaction information is sent to the sub server apparatus300.

FIGS.11A to11Dare diagrams illustrating the relationships between the audience seats in the virtual audience seat space5illustrated inFIG.8Band the displayed image.FIG.11Ais a conceptual diagram illustrating an example of the positional relationships between the virtual stage space4and the audience seats in the virtual audience seat space5, andFIG.11Bis a diagram illustrating an example of an image and audio when the content distributed from the main server apparatus200is viewed from the viewpoint of the audience seat E selected by the viewing apparatus400.FIG.11Cis a diagram illustrating an example of an image and audio based on environment data generated taking the audience seat E selected by the viewing apparatus400in the virtual audience seat space5distributed from the sub server apparatus300as the position of the viewpoint, andFIG.11Dis a diagram illustrating an example of an image and audio obtained by compositing those illustrated inFIGS.11B and11C.

In this case, as illustrated inFIG.11D, the avatar of the viewing apparatus400located at the adjacent audience seat D is displayed so as to enter the field of view, and the audio of the neighboring audience seats D, I, and J is output by compositing that audio with the audio of the content. In addition, when configuring the environment image data, the sub server apparatus300sets a restriction on the movement of the avatars such that the avatars of the plurality of viewing apparatuses400in the virtual audience seat space5do not share the same three-dimensional coordinates, with the exception of the outlines of the avatars, and adjusts the position of the viewpoint. Additionally, when the outlines of a plurality of avatars share the same three-dimensional coordinates, a sound expressing contact may be generated and composited with the environment audio data.

When generating the environment data, the sub server apparatus300generates the haptics data, which provides a sense of contact as between users in a physical space, for the viewing apparatuses400, among the plurality of viewing apparatuses400for the avatars sharing the same coordinate positions, that have the haptics device414.

FIGS.12A to12Dare diagrams illustrating the relationship between another audience seat in the cyber space illustrated inFIG.8Band the displayed image.FIG.12Ais a conceptual diagram illustrating an example of the positional relationship between the virtual stage space4and the audience seats in the virtual audience seat space5, and illustrates a case where the viewpoint of the viewing apparatus400is positioned at audience seat H. Here, the image and audio for a case where the content distributed from the main server apparatus200is viewed from the viewpoint of the audience seat H are indicated inFIG.12B, and the image and audio based on the environment data generated using the audience seat H as the viewpoint, distributed from the sub server apparatus300, are indicated inFIG.12C. In this case, an image obtained by compositing those ofFIGS.12B and12Cis displayed such that the avatars of the viewing apparatuses400from the audience seats G, B, C, D, and I, which are in front of and to the left and right of the audience seat H, enter the field of view, and the audio from the audience seats G, B, C, D, and I around the audience seat His output having been composited with the audio of the content. If the viewing apparatus400of the audience seat H includes the haptics device414, haptics data that conveys a sense of contact between the avatars is output, in the same manner as in the example described above.

As described above, according to the first embodiment, when viewing distributed content, information on other viewers viewing the same content can be composited and expressed with the content to obtain a greater sense of immersion.

Additionally, the haptics device can be used to express contact with other viewers to provide an even greater sense of immersion.

The foregoing has described an example in which the main server apparatus200distributes the 3D spatial data with audio having added thereto parameters for allowing only the generation of the image data and the audio data from the viewpoint of each viewing apparatus400. However, the present disclosure is not limited thereto, and the main server apparatus200may generate the image data and audio data from the viewpoint of each viewing apparatus400to which the data is distributed from the 3D spatial data with audio, and distribute the generated data. In this case, the viewpoint of each of the viewing apparatuses400calculated in the sub server apparatus300is obtained as necessary.

Although the foregoing describes the haptics device and haptics data as being related to the sense of touch, the present disclosure is not limited thereto, and the haptics device and haptics data may be related to any sensory information aside from vision and hearing. Aside from tactile information, the sensory information may be taste information or smell information, for example.

Variation

Although the foregoing first embodiment described a case where the environment image is rendered in the sub server apparatus300, the environment image may be rendered in the viewing apparatus400. In this case, the avatar information and reaction information from the audience seats in the field of view is received from the sub server apparatus300based on the positional relationship between the viewpoint of the viewing apparatus400and the virtual stage space4. Then, in the viewing apparatus400, the environment image data and the environment audio data are generated, composited with the distributed content, and output. If the viewing apparatus400includes the haptics device414, the viewing apparatus400generates the haptics data and drives the haptics device414.

FIG.6Billustrates an example of the avatar information and reaction information received from the sub server apparatus300in the present variation.

According to the present variation as described thus far, not only can the same effects as those of the first embodiment be achieved, but it is no longer necessary for the sub server apparatus300to generate environment data for all of the viewing apparatuses400to which content is to be distributed, and the load on the sub server apparatus300can therefore also be reduced.

Second Embodiment

A second embodiment of the present disclosure will be described next.

The first embodiment described a case where the environment data output from the sub server apparatus300is composited with the content output from the main server apparatus200and then output in order to achieve a sense of immersion. However, depending on the viewer, the viewing environment, and the like, the viewer may wish to enjoy the content distributed from the main server apparatus200as-is. The second embodiment will describe control for not outputting environment images, environment audio, and haptics based on the environment data in such a case.

The configurations of the main server apparatus200, the sub server apparatus300, and the viewing apparatus400, as well as the processing by the main server apparatus200and the sub server apparatus300, are the same as those described in the first embodiment above, and will therefore not be described here.

FIG.13is a flowchart illustrating viewing processing performed by the viewing apparatus400according to the second embodiment. Here, processing for determining whether to turn off the output of the environment data output from the sub server apparatus300has been added to the processing illustrated inFIG.10. Note that steps that are the same as those illustrated inFIG.10will be given the same reference signs, and descriptions thereof will be omitted as appropriate.

If the content is being distributed (NO in step S305), the content is received from the main server apparatus200in step S306, after which it is determined, in step S320, whether an instruction to turn off the output of the environment data sent from the sub server apparatus300has been made.

Note that various methods are conceivable as methods for turning the output of the environment data on and off. For example, a physical switch for turning the output of the environment data on and off may be provided, or a software button rendered on the screen of the display unit405may be turned on and off using a program by operating the touch panel, an operation member, or the like.

If an instruction to turn the output of the environment data off has not been made, the sequence moves to step S307, whereas if an instruction to turn the output of the environment data off has been made, the sequence moves to step S321. Then, the image data and the audio data from the current viewpoint are generated from the content distributed from the main server apparatus200, the generated image data and audio data are output, and the sequence moves to step S309.

If the output of the environment data sent from the sub server apparatus300is turned off in this manner, in the example illustrated inFIGS.11A to11D, the image illustrated inFIG.11Bwill be displayed, whereas in the example illustrated inFIGS.12A to12D, the image illustrated inFIG.12Bwill be displayed.

According to the second embodiment as described thus far, the output of the environment data distributed from the sub server apparatus300can be turned on and off in accordance at the viewer's discretion, which makes it possible for the viewer to view the content according to their preferences, the state of other avatars, and the like.

Note that a fee may be charged to turn the output of the environment data off, and in this case, a notification that a fee will be charged to turn the environment data off is issued to the viewing apparatus400. If the viewer of the viewing apparatus400accepts the fee, a procedure for charging the fee is performed, after which control is performed such that the environment data is not distributed from the sub server apparatus300to the viewing apparatus400.

Third Embodiment

A third embodiment of the present disclosure will be described next.

The first and second embodiments described cases where the audience seat of the viewing apparatus400in the virtual audience seat space5is determined in advance. The third embodiment, however, will describe a case where movement is possible within the virtual audience seat space5.

The configurations of the main server apparatus200, the sub server apparatus300, and the viewing apparatus400, as well as the processing by the main server apparatus200and the viewing apparatus400, are the same as those described in the first embodiment above, and will therefore not be described here.

FIGS.14A and14Bshow a flowchart illustrating processing performed by the sub server apparatus300according to the third embodiment.

First, in step S201, the apparatus stands by for the information on the configuration of the cyber space sent from the main server apparatus200in step S104. Once the information on the configuration of the cyber space is received, the sequence moves to step S202, and the virtual audience seat space5is generated. Here, it is assumed that the virtual audience seat space5is divided into a plurality of regions, as illustrated inFIG.15A.

When the calculation of the configuration of the virtual audience seat space5in step S202ends, in step S203, the main server apparatus200searches for the viewing apparatuses400that wish to view the content before the content distribution is started. Information such as the avatar information, the IP address, and the like of the viewing apparatus400that wishes to view the content is then obtained.

Next, in step S231, a viewing position of the viewing apparatus400is set to a desired position in the virtual audience seat space5that does not overlap with the viewing positions of the other viewing apparatuses400, and a congestion map is generated.FIG.15Bis a diagram illustrating an example of the congestion map, and illustrates an example of divided regions in the virtual audience seat space5and the total number of viewing apparatuses400having viewing positions within each divided region. The sub server apparatus300counts the number of viewing positions of the viewing apparatuses400in each divided region and stores that number in association with the corresponding divided region.

Once the congestion map is generated, in step S204, the IP addresses of the viewing apparatuses400that wish to view the content are sent to the main server apparatus200.

Note that a fee may be charged when an application to view the content is made, and the amount charged may differ depending on the location of the audience seat.

Next, in step S205, it is determined whether or not the main server apparatus200has started distributing the content. If so, the sequence moves to step S206, and if not, the sequence returns to step S203and the processing is repeated.

In step S206, it is determined whether the main server apparatus200has ended the distribution of the content, and if so, the processing ends. However, if the distribution has not ended, the sequence moves to step S207, where the reaction information is obtained from the viewing apparatuses400to which the content is being distributed.

Then, in step S232, whether the viewer of the viewing apparatus400has moved between divided regions is determined based on the change amount information in the reaction information obtained in step S207. If not, the sequence moves to step S208, but if so, the sequence moves to step S233.

In step S233, it is determined, based on the congestion map, whether the counted number of the divided region in the direction in which the viewer moved is greater than a predetermined first threshold Th1. If the number is greater than the first threshold Th1, in step S234, the movement between the divided regions in the virtual audience seat space5is disabled such that the viewer does not cross into the other divided region, and a notification indicating that movement in that direction is not possible is issued to the viewing apparatus400, after which the sequence moves to step S208. At this time, guidance such as “you cannot move in that direction” may be provided as text or audio, or an image showing a crowd of avatars too thick to move through may be generated to present a visual indication that movement in that direction is impossible, a sound indicating that avatars are in contact with each other may be generated and output, or the like.

Furthermore, the sub server apparatus300may generate and output the haptics data, which provides a sense of contact as between users in a physical space, to the viewing apparatus400having the haptics device414. Alternatively, haptics data may be generated to provide a sense of coming into contact with a predetermined hard, flat surface, such as a wall, in the direction in which movement is not possible.

On the other hand, if the number is not greater than the predetermined first threshold Th1, the sequence moves to step S235, where it is determined whether the counted number in the divided region in the direction of movement is greater than a predetermined second threshold Th2higher than the first threshold Th1. If the number is not greater than the second threshold Th2, the sequence moves to step S208. However, if the number is greater than the second threshold Th2, the sequence moves to step S236, where a notification indicating that the destination divided region is crowded is issued, after which the sequence moves to step S208. At this time, guidance such as “the area is too crowded” may be provided as text or audio, or an image showing a thick crowd of avatars may be generated to present a visual indication of the crowding, a sound indicating that the avatars are in contact with each other may be generated and output, or the like.

Furthermore, the sub server apparatus300may generate and output the haptics data, which provides a sense of contact as between users in a physical space, to the viewing apparatus400having the haptics device414.

In step S208, it is determined whether the environment data has been generated for each of the viewing apparatuses400to which the content is being distributed. If there is a viewing apparatus400for which the environment data has not been generated, in step S209, the environment data for that viewing apparatus400is generated, after which the sequence returns to step S232.

However, if the environment data has been generated for all the viewing apparatuses400to which the content is being distributed, the environment data generated in step S210is output to each viewing apparatus400. The number of viewing positions set in each divided region is then counted in step S237, the congestion map is updated, and the sequence returns to step S206. Note that the timing at which the environment data is output is not limited thereto, and the environment data may be output to each corresponding viewing apparatus400in sequence each time the environment data is generated in step S209.

According to the third embodiment described thus far, a greater sense of immersion can be achieved in a situation where movement within the virtual audience seat space5is possible, such as during an outdoor concert in a physical space, for example.

Variation

A desired fee may be set for the divided regions into which the virtual audience seat space5is divided. In this case, if, when the motion information indicates movement between divided regions, the counted number for the divided region being moved to is not greater than the first threshold Th1and the fee will change, a notification to that effect may be issued to the viewing apparatus400. If the viewer of the viewing apparatus400accepts the fee change, a procedure for charging the fee is performed, the motion information is enabled, and the viewer is then able to move between the divided regions based on the motion information. On the other hand, if the viewer of the viewing apparatus400does not accept the fee change, the motion information is disabled, and the viewer is prevented from moving between the divided regions based on the motion information.

The first threshold Th1may also be set for each divided region. For example, it is conceivable to use a setting method which sets a higher fee, and reduces the first threshold Th1to provide a better viewing environment, for divided regions that are closer to the virtual stage space4.

Additionally, a divided region in which only viewers who are members can establish viewing positions may be set in advance, for example.

In this manner, the distributor of the content may be capable of setting various conditions as appropriate, such as fees that allow viewing positions to be set in divided regions.

Other Embodiments

Although the foregoing embodiments described a single virtual audience seat space5as being provided for the virtual stage space4, the present disclosure is not limited thereto, and a plurality of virtual audience seat spaces5may be provided for a single virtual stage space4. In this case, the viewpoint, viewing position, environment data, and the like of the viewing apparatus400may be controlled for each virtual audience seat space5.

The foregoing embodiments have described the main server apparatus200as distributing the content and the sub server apparatus300as generating and sending the environment data. However, the present disclosure is not limited thereto, and the content may be distributed, and the environment data generated and sent, by a single server apparatus.

This application claims the benefit of priority from Japanese Patent Application No. 2023-035860, filed Mar. 8, 2023, and Japanese Patent Application No. 2023-205438, filed Dec. 5, 2023, which are hereby incorporated by reference herein in their entirety.