Patent ID: 12200181

DETAILED DESCRIPTION

An example of an embodiment according to the technology of the present disclosure will be described with reference to the accompanying drawings.

First, the terms used in the following description will be described.

CPU refers to an abbreviation of “central processing unit”. RAM refers to an abbreviation of “random access memory”. DRAM refers to an abbreviation of “dynamic random access memory”. SRAM refers to an abbreviation of “static random access memory”. ROM refers to an abbreviation of “read only memory”. SSD refers to an abbreviation of “solid state drive”. HDD refers to an abbreviation of “hard disk drive”. EEPROM refers to an abbreviation of “electrically erasable and programmable read only memory”. I/F refers to an abbreviation of “interface”. IC refers to an abbreviation of “integrated circuit”. ASIC refers to an abbreviation of “application specific integrated circuit”. PLD refers to an abbreviation of “programmable logic device”. FPGA refers to an abbreviation of “field-programmable gate array”. SoC refers to an abbreviation of “system-on-a-chip”. CMOS refers to an abbreviation of “complementary metal oxide semiconductor”. CCD refers to an abbreviation of “charge coupled device”. EL refers to an abbreviation of “electro-luminescence”. GPU refers to an abbreviation of “graphics processing unit”. LAN refers to an abbreviation of “local area network”. 3D refers to an abbreviation of “3 dimension”. USB refers to an abbreviation of “universal serial bus”. In addition, in the description of the present specification, the meaning of “vertical” includes the meaning of a perfect vertical as well as the meaning of a substantially vertical including errors allowed in design and manufacturing. In addition, in the description of the present specification, the meaning of “match” includes the meaning of a perfect match as well as the meaning of a substantially match including errors allowed in design and manufacturing.

For example, as shown inFIG.1, an information processing system10comprises an information processing apparatus12, a smart device14, a plurality of imaging apparatuses16, an imaging apparatus18, a wireless communication base station (hereinafter, simply referred to as “base station”)20, and a receiver34. Note that, here, the “smart device14” refers to a portable multifunctional terminal, such as a smartphone or a tablet terminal. Here, the smart device14is an example of a “first display device” according to the technology of the present disclosure, the imaging apparatus18is an example of a “first imaging apparatus” and a “second imaging apparatus” according to the technology of the present disclosure, and the receiver34is an example of a “second display device” according to the technology of the present disclosure. Note that, here, although the receiver34is described as an example, the technology of the present disclosure is not limited to this, and may be an electronic device with a display (for example, a smart device). In addition, the number of the base stations20is not limited to one, and a plurality of the base stations20may be present. Further, the communication standards used in the base station include a wireless communication standard including a Long Term Evolution (LTE) standard and a wireless communication standard including a WiFi (802.11) standard and/or a Bluetooth (registered trademark) standard.

The imaging apparatuses16and18are devices for imaging having a CMOS image sensor, and each have an optical zoom function and/or a digital zoom function. Note that another type of image sensor, such as a CCD image sensor, may be adopted instead of the CMOS image sensor. Hereinafter, for convenience of description, in a case in which a distinction is not necessary, the imaging apparatus18and the plurality of imaging apparatuses16, are referred to as “plurality of imaging apparatuses” without reference numeral.

The plurality of imaging apparatuses16are installed in a soccer stadium22. Each of the plurality of imaging apparatuses16is disposed so as to surround a soccer field24, and images a region including the soccer field24as an imaging region. Here, an aspect example is described in which each of the plurality of imaging apparatuses16is disposed so as to surround the soccer field24. However, the technology of the present disclosure is not limited to this, and the disposition of the plurality of imaging apparatuses16is decided depending on a virtual viewpoint video requested to be generated by the viewer28or the like. The plurality of imaging apparatuses16may be disposed so as to surround the whole soccer field24, or the plurality of imaging apparatuses16may be disposed so as to surround a specific part thereof. The imaging apparatus18is installed in an unmanned type aerial vehicle (for example, a multi rotorcraft type unmanned aerial vehicle), and images the region including the soccer field24as the imaging region in a bird's-eye view from the sky. The imaging region of the region including the soccer field24in a bird's-eye view from the sky refers to an imaging face on the soccer field24by the imaging apparatus18.

The information processing apparatus12is installed in a control room32. The plurality of imaging apparatuses16and the information processing apparatus12are connected to each other via a LAN cable30, and the information processing apparatus12controls the plurality of imaging apparatuses16and acquires an image obtained by being imaged by each of the plurality of imaging apparatuses16. Note that although the connection using a wired communication method by the LAN cable30is described as an example here, the technology of the present disclosure is not limited to this, and the connection using a wireless communication method may be used.

In the soccer stadium22, spectator seats26are provided so as to surround the soccer field24, and the viewer28sits in the spectator seat26. The viewer28owns the smart device14, and the smart device14is used by the viewer28. Note that, here, an aspect example is described in which the viewer28is present in the soccer stadium22, but the technology of the present disclosure is not limited to this, and the viewer28may be present outside the soccer stadium22.

The base station20transmits and receives various pieces of information to and from the information processing apparatus12and the unmanned aerial vehicle27via radio waves. That is, the information processing apparatus12is connected to the unmanned aerial vehicle27via the base station20in the wirelessly communicable manner. The information processing apparatus12controls the unmanned aerial vehicle27by wirelessly communicating with the unmanned aerial vehicle27via the base station20, and acquires the image obtained by being imaged by the imaging apparatus18from the unmanned aerial vehicle27.

The base station20transmits various pieces of information to the receiver34via the wireless communication. The information processing apparatus12transmits various videos to the receiver34via the base station20, the receiver34receives various videos transmitted from the information processing apparatus12, and the received various videos are displayed on a screen34A. Note that the receiver34is used for viewing by an unspecified number of spectators, for example. A location in which the receiver34is installed may be inside the soccer stadium22or outside the soccer stadium22(for example, a public viewing venue). Note that, here, although an aspect example is described in which various pieces of information is transmitted to the receiver34via the wireless communication, the technology of the present disclosure is not limited to this, and for example, an aspect may be adopted in which various pieces of information is transmitted to the receiver34via a wired communication.

The information processing apparatus12is a device corresponding to a server, and the smart device14is a device corresponding to a client terminal with respect to the information processing apparatus12. By the information processing apparatus12and the smart device14wirelessly communicating with each other via the base station20, the smart device14requests the information processing apparatus12to provide various services, and the information processing apparatus12provides the services to the smart device14in response to the request from the smart device14.

For example, as shown inFIG.2, the information processing apparatus12acquires a bird's-eye view video46A showing the region including the soccer field24in a case of being observed from the sky from the unmanned aerial vehicle27. The bird's-eye view video46A is a moving image obtained by imaging the region including the soccer field24as the imaging region (hereinafter, also simply referred to as “imaging region”) in a bird's-eye view from the sky by the imaging apparatus18of the unmanned aerial vehicle27. Note that, here, although a case in which the bird's-eye view video46A is the moving image is described as an example, the bird's-eye view video46A is not limited to this, and may be a still image showing the region including the soccer field24in a case of being observed from the sky.

The information processing apparatus12acquires an imaging video46B showing the imaging region in a case of being observed from each position of the plurality of imaging apparatuses16from each of the plurality of imaging apparatuses16. The imaging video46B is a moving image obtained by imaging the imaging region by each of the plurality of imaging apparatuses16. Note that, here, although a case in which the imaging video46B is the moving image is described as an example, the imaging video46B is not limited to this, and may be a still image showing the imaging region in a case of being observed from each position of the plurality of imaging apparatuses16.

The information processing apparatus12generates a virtual viewpoint video46C based on the bird's-eye view video46A and the imaging video46B. The virtual viewpoint video46C is a video showing the imaging region in a case in which the imaging region is observed from a specific viewpoint position and a specific visual line direction. In the example shown inFIG.2, the virtual viewpoint video46C refers to, for example, the virtual viewpoint video showing the imaging region in a case in which the imaging region is observed from a viewpoint position42and a visual line direction44in a spectator seat26. Examples of the virtual viewpoint video46C include a moving image using 3D polygons. The viewpoint position42and the visual line direction44are not fixed. That is, the viewpoint position42and the visual line direction44are changed in response to an instruction from the viewer28or the like. Here, an aspect example is described in which the viewpoint position42and the visual line direction44are not fixed, but the technology of the present disclosure is not limited to this, and the viewpoint position42and the visual line direction44may be fixed.

The information processing apparatus12generates the moving image using the 3D polygons by composing the plurality of images (here, for example, the bird's-eye view video46A and the imaging video46B) obtained by imaging the imaging region by the plurality of imaging apparatuses. The information processing apparatus12generates the virtual viewpoint video corresponding to a case in which the imaging region is observed from any position and any direction based on the moving image using the generated 3D polygons. In the present embodiment, the information processing apparatus12generates the virtual viewpoint video46showing the subject in a case in which the subject is observed from the viewpoint position42and the visual line direction44. Stated another way, the virtual viewpoint video46refers to the video corresponding to the video obtained by imaging by a virtual imaging apparatus (hereinafter, also referred to as “virtual imaging apparatus”) installed at the viewpoint position42with the visual line direction44as the imaging direction. Here, the moving image is described as an example of the virtual viewpoint video46C, but the technology of the present disclosure is not limited to this, and a still image may be used. Note that the viewpoint position42is an example of a “specific viewpoint position” according to the technology of the present disclosure, and the visual line direction44is an example of a “specific visual line direction” according to the technology of the present disclosure. In addition, the virtual viewpoint video46C is an example of a “free viewpoint video” according to the technology of the present disclosure.

In addition, here, an aspect example is described in which the bird's-eye view video46A obtained by being imaged by the imaging apparatus18is also used for generation, but the technology of the present disclosure is not limited to this. For example, the bird's-eye view video46A is not provided for generation of the virtual viewpoint video46C, and only a plurality of the imaging videos46B obtained by being imaged by the plurality of imaging apparatuses16may be used for generating the virtual viewpoint video46C. That is, the virtual viewpoint video46C may be generated only from the videos obtained by being imaged by the plurality of imaging apparatuses16without using the video obtained by the imaging apparatus18(for example, a multi rotorcraft type unmanned aerial vehicle). Note that in a case in which the video obtained from the imaging apparatus18(for example, a multi rotorcraft type unmanned aerial vehicle) is used, a more accurate virtual viewpoint video can be generated.

The information processing apparatus12selectively transmits a reference video46B1(seeFIGS.9and10), which will be described below, and the virtual viewpoint video46C to the smart device14and the receiver34as the distribution video. Note that, in the following, for convenience of description, in a case in which a distinction is not necessary, the reference video46B1and the virtual viewpoint video46C are also referred to as the “distribution video”.

For example, as shown inFIG.3, the information processing apparatus12comprises a computer50, a reception device52, a display53, a first communication I/F54, and a second communication I/F56. The computer50comprises a CPU58, a storage60, and a memory62, and the CPU58, the storage60, and the memory62are connected to each other via a bus line64. In the example shown inFIG.3, for convenience of illustration, one bus line is shown as the bus line64, but a plurality of bus lines may be used. In addition, the bus line64may include a serial bus or a parallel bus configured by a data bus, an address bus, a control bus, and the like.

The CPU58controls the whole information processing apparatus12. Various parameters and various programs are stored in the storage60. The storage60is a non-volatile storage device. Here, a flash memory is adopted as an example of the storage60, but the technology of the present disclosure is not limited to this, and an EEPROM, an HDD, an SSD, or the like may be used. The memory62is a storage device. Various pieces of information are transitorily stored in the memory62. The memory62is used as a work memory by the CPU58. Here, a DRAM is adopted as an example of the memory62, but the technology of the present disclosure is not limited to this, and another type of storage device may be used.

The reception device52receives the instruction from a user or the like of the information processing apparatus12. Examples of the reception device52include a touch panel, a hard key, and a mouse. The reception device52is connected to the bus line64and the like, and the CPU58acquires the instruction received by the reception device52.

The display53is connected to the bus line64and displays various pieces of information under the control of the CPU58. Examples of the display53include a liquid crystal display. Note that another type of display, such as an organic EL display or an inorganic EL display, may be adopted as the display53without being limited to the liquid crystal display.

The first communication I/F54is connected to the LAN cable30. The first communication I/F54is realized by a device having an FPGA, for example. The first communication I/F54is connected to the bus line64and controls the exchange of various pieces of information between the CPU58and the plurality of imaging apparatuses16. For example, the first communication I/F54controls the plurality of imaging apparatuses16in response to the request of the CPU58. In addition, the first communication I/F54acquires the imaging video46B (seeFIG.2) obtained by being imaged by each of the plurality of imaging apparatuses16, and outputs the acquired imaging video46B to the CPU58. Note that, here, although the wired communication I/F is described as an example of the first communication I/F54, a wireless communication I/F, such as a high-speed wireless LAN, may be used.

The second communication I/F56is connected to the base station20in the wirelessly communicable manner. The second communication I/F56is realized by a device having an FPGA, for example. The second communication I/F56is connected to the bus line64. The second communication I/F56controls the exchange of various pieces of information between the CPU58and the unmanned aerial vehicle27by the wireless communication method via the base station20. In addition, the second communication I/F56controls the exchange in various pieces of information between the CPU58and the smart device14by the wireless communication method via the base station20. In addition, the second communication I/F56controls the transmission of various videos to the receiver34by the CPU58by the wireless communication method via the base station20. Note that at least one of the first communication I/F54or the second communication I/F56can be configured by a fixed circuit instead of an FPGA. In addition, at least one of the first communication I/F54or the second communication I/F56may be a circuit configured by an ASIC, an FPGA, and/or a PLD or the like.

For example, as shown inFIG.4, the smart device14comprises a computer70, a gyro sensor74, a reception device76, a display78, a microphone80, a speaker82, an imaging apparatus84, and a communication I/F86. The computer70comprises a CPU88, a storage90, and a memory92, and the CPU88, the storage90, and the memory92are connected to each other via a bus line94. In the example shown inFIG.4, for convenience of illustration, one bus line is shown as the bus line94. However, the bus line94is configured by a serial bus or is configured to include a data bus, an address bus, a control bus, and the like. In addition, in the example shown inFIG.4, the CPU88, the reception device76, the display78, the microphone80, the speaker82, the imaging apparatus84, and the communication I/F86are connected by a common bus, but the CPU88and each device may be connected by a dedicated bus or a dedicated communication line.

The CPU88controls the whole smart device14. Various parameters and various programs are stored in the storage90. The storage90is a non-volatile storage device. Here, a flash memory is adopted as an example of the storage90, but the technology of the present disclosure is not limited to this, and an EEPROM, an HDD, an SSD, or the like may be used. Various pieces of information are transitorily stored in the memory92, and the memory92is used as a work memory by the CPU88. Here, an RAM is adopted as an example of the memory92, but the technology of the present disclosure is not limited to this, and another type of storage device may be used.

The gyro sensor74measures an angle around a yaw axis of the smart device14(hereinafter, also referred to as “yaw angle”), an angle around a roll axis of the smart device14(hereinafter, also referred to as “roll angle”), and an angle around a pitch axis of the smart device14(hereinafter, also referred to as “pitch angle”). The gyro sensor74is connected to the bus line94, and angle information indicating the yaw angle, the roll angle, and the pitch angle measured by the gyro sensor74is acquired by the CPU88via the bus line94and the like.

The reception device76is an example of a “reception unit (acceptor)” according to the technology of the present disclosure, and receives the instruction from the user or the like of the smart device14(here, for example, the viewer28). Examples of the reception device76include a touch panel76A, and a hard key. The reception device76is connected to the bus line94, and the CPU88acquires the instruction received by the reception device76.

The display78is connected to the bus line94and displays various pieces of information under the control of the CPU88. Examples of the display78include a liquid crystal display. Note that another type of display, such as an organic EL display, may be adopted as the display78without being limited to the liquid crystal display.

The smart device14comprises a touch panel display, and the touch panel display is realized by the touch panel76A and the display78. That is, the touch panel display is formed by superimposing the touch panel76A on a display region of the display78or by building a touch panel function in the display78(“in-cell” type).

The microphone80converts a collected sound into an electric signal. The microphone80is connected to the bus line94. The CPU88acquires the electric signal obtained by converting the sound collected by the microphone80via the bus line94.

The speaker82converts the electric signal into the sound. The speaker82is connected to the bus line94. The speaker82receives the electric signal output from the CPU88via the bus line94, converts the received electric signal into the sound, and outputs the sound obtained by converting the electric signal to the outside of the smart device14.

The imaging apparatus84acquires an image showing a subject by imaging the subject. The imaging apparatus84is connected to the bus line94. The image obtained by imaging the subject by the imaging apparatus84is acquired by the CPU88via the bus line94. Here, the speaker82is integrated with the smart device14, but the sound output by a separate headphone (including earphones) connected to the smart device14by wire or wirelessly may be adopted.

The communication I/F86is connected to the base station20in the wirelessly communicable manner. The communication I/F86is realized by, for example, a device configured by circuits (for example, an ASIC, an FPGA, and/or a PLD). The communication I/F86is connected to the bus line94. The communication I/F86controls the exchange of various pieces of information between the CPU88and an external device by the wireless communication method via the base station20. Here, examples of the “external device” include the information processing apparatus12.

For example, as shown inFIG.5, the roll axis is an axis that passes through a center of the display78of the smart device14. A rotation angle of the smart device14around the roll axis is measured by the gyro sensor74as the roll angle. In addition, as shown inFIG.6, for example, the yaw axis is an axis that passes through a center of a lateral peripheral surface of the smart device14in a longitudinal direction of the lateral peripheral surfaces. A rotation angle of the smart device14around the yaw axis is measured by the gyro sensor74as the yaw angle. Further, as shown inFIG.7, for example, the pitch axis is an axis that passes through a center of the lateral peripheral surface of the smart device14in a lateral direction of the lateral peripheral surfaces. The rotation angle of the smart device14around the pitch axis is measured by the gyro sensor74as the pitch angle.

For example, as shown inFIG.8, in the information processing apparatus12, the storage60stores an information processing program60A, a display control program60B, and a setting program60C. Note that, in the following, in a case in which a distinction is not necessary, the information processing program60A, the display control program60B, and the setting program60C are referred to as an “information processing apparatus side program” without reference numeral.

The CPU58is an example of a “processor” according to the technology of the present disclosure, and the memory62is an example of a “memory” according to the technology of the present disclosure. The CPU58reads out the information processing apparatus side program from the storage60, and expands the readout information processing apparatus side program in the memory62. The CPU58performs the exchange in various pieces of information between the smart device14, the imaging apparatus16, and the unmanned aerial vehicle27according to the information processing apparatus side program expanded in the memory62, and performs transmission of various videos to the receiver34.

The CPU58reads out the information processing program60A from the storage60, and expands the readout information processing program60A in the memory62. The CPU58is operated as an acquisition unit102and an execution unit104according to the information processing program60A expanded in the memory62. The CPU58is operated as the acquisition unit102and the execution unit104to execute information processing (seeFIG.21), which will be described below.

The CPU58reads out the display control program60B from the storage60, and expands the readout display control program60B in the memory62. The CPU58is operated as a control unit106according to the display control program60B expanded in the memory62to execute a display control process (seeFIG.19) described below.

The CPU58reads out the setting program60C from the storage60, and expands the readout setting program60C in the memory62. The CPU58is operated as a setting unit108according to the setting program60C expanded in the memory62. The CPU58is operated as the setting unit108to execute a setting process (seeFIG.20), which will be described below.

For example, as shown inFIG.9, a video generation program60D is stored in the storage60, and the CPU58reads out the video generation program60D from the storage60and expands the readout video generation program60D in the memory62. The CPU58is operated as a reference video generation unit100A and a virtual viewpoint video generation unit100B according to the video generation program60D expanded in the memory62. The reference video generation unit100A generates the reference video46B1, and the virtual viewpoint video generation unit100B generates the virtual viewpoint video46C.

For example, as shown inFIG.10, in the information processing apparatus12, imaging video designation information is received by the reception device52. The imaging video designation information refers to information for designating one of a plurality of the imaging videos. The imaging video designation information is received by the reception device52, for example. The reception of the imaging video designation information by the reception device52is realized, for example, by operating the reception device52by a video creator (not shown). Specifically, in a state in which the plurality of imaging videos46B are displayed side by side on the display53of the information processing apparatus12, one of the plurality of imaging videos is designated according to the imaging video designation information received by the reception device52by the video creator.

The reference video generation unit100A generates the reference video46B1by receiving the plurality of imaging videos from the imaging apparatus18and the plurality of imaging apparatuses16, acquiring the imaging video designated by the imaging video designation information from the received plurality of imaging videos, and performing various pieces of signal processing (for example, known signal processing) on the acquired imaging video46B. That is, the reference video46B1is the video obtained by imaging the imaging region by an imaging apparatus (hereinafter, also referred to as a “reference imaging apparatus”) of any of the plurality of imaging apparatuses16. Here, the “reference video46B1” is, for example, a live broadcast video, a video imaged in advance, or the like. The live broadcast video or the video imaged in advance is, for example, a video broadcast on a television or distributed on the Internet. Note that, here, the live broadcast video is generated based on the imaging video46B indicated by imaging video indication information, but it is merely an example, and the imaging video46B itself designated by the imaging video designation information may be adopted as the reference video46B1.

For example, as shown inFIG.11, the touch panel76A of the smart device14receives a viewpoint visual line indication. In this case, for example, first, the distribution video from the information processing apparatus12is received by the smart device14. Next, the distribution video received from the information processing apparatus12by the smart device14is displayed on the display78. In a state in which the distribution video is displayed on the display78, the viewpoint visual line indication is received by the touch panel76A by operating the touch panel76A by the viewer28.

The viewpoint visual line indication includes a viewpoint position indication which is an indication of the viewpoint position42(seeFIG.2) with respect to the imaging region and a visual line direction indication which is an indication of the visual line direction44(seeFIG.2) with respect to the imaging region. In the present embodiment, the viewpoint position indication is first performed, and then the visual line direction indication is performed. That is, first, the viewpoint position42is decided, and then the visual line direction44is decided.

Examples of the viewpoint position indication include a tap operation on the touch panel76A. In addition, the tap operation may be a single tap operation or a double tap operation. In addition, instead of the tap operation, a long touch operation may be performed. Examples of the visual line direction indication include a slide operation. Here, the “slide operation” refers to, for example, an operation of linearly sliding the position in which an indicator (for example, a finger of the viewer28) is in contact with the touch panel76A. In this manner, the position at which the tap operation is performed on the touch panel76A corresponds to the viewpoint position42with respect to the imaging region, and the direction in which the slide operation is performed on the touch panel76A corresponds to the visual line direction44with respect to the imaging region.

The virtual viewpoint video generation unit100B generates the virtual viewpoint video46C by acquiring the viewpoint visual line indication received by the touch panel76A, and using the plurality of imaging videos in response to the acquired viewpoint visual line indication. That is, the virtual viewpoint video generation unit100B generates the virtual viewpoint video46C showing the imaging region in a case in which the imaging region is observed from the viewpoint position42in response to the viewpoint position indication and the visual line direction44in response to the visual line direction indication. The generation of the virtual viewpoint video46C is realized, for example, by generating the 3D polygons based on the plurality of imaging videos.

For example, as shown inFIG.12, in a case in which the viewpoint visual line indication is changed, that is, in a case in which the viewpoint position42and the visual line direction44are changed, an aspect of the virtual viewpoint video46C, that is, a size, a direction, and the like of the subject shown in the virtual viewpoint video46are changed. Note that the enlargement of the virtual viewpoint video46C is realized, for example, according to a pinch-out operation on the touch panel76A of the smart device14, and the reduction of the virtual viewpoint video46C is realized, for example, according to a pinch-in operation on the touch panel76A of the smart device14.

For example, as shown inFIG.13, the acquisition unit102acquires the reference video46B1generated by the reference video generation unit100A from the reference video generation unit100A. In addition, the acquisition unit102acquires the virtual viewpoint video46C generated by the virtual viewpoint video generation unit100B from the virtual viewpoint video generation unit100B.

The control unit106includes a reference video control unit106A and a virtual viewpoint video control unit106B. The reference video control unit106A controls the receiver34such that the reference video46B1is displayed on the screen34A by transmitting the reference video46B1acquired by the acquisition unit102to the receiver34, which is a display device different from the display78of the smart device14. That is, the receiver34receives the reference video46B1transmitted from the reference video control unit106A, and displays the received reference video46B1on the screen34A.

The virtual viewpoint video control unit106B controls the smart device14such that the virtual viewpoint video46C is displayed on the display78by transmitting the virtual viewpoint video46C acquired by the acquisition unit102to the smart device14. That is, the smart device14receives the virtual viewpoint video46C transmitted from the virtual viewpoint video control unit106B, and displays the received virtual viewpoint video46C on the display78.

For example, as shown inFIG.14, the execution unit104executes a specific process based on the reference video46B1acquired by the acquisition unit102and the virtual viewpoint video46C acquired by the acquisition unit102. Here, the specific process is a process performed in a case in which a difference degree between the reference video46B1acquired by the acquisition unit102and the virtual viewpoint video46C acquired by the acquisition unit102(hereinafter, simply referred to as the “difference degree”) is equal to or more than a first threshold value. Examples of the specific process include a process that contributes to setting of the difference degree to be less than the first threshold value. In addition, the execution unit104executes the specific process depending on the change in the virtual viewpoint video46C. That is, the execution unit104executes the specific process while following the change in the virtual viewpoint video46C.

For example, as shown inFIG.15, the execution unit104includes a difference degree calculation unit104A, a determination unit104B, and a notification processing unit104C. The difference degree calculation unit104A calculates the difference degree between the reference video46B1acquired by the acquisition unit102and the virtual viewpoint video46C acquired by the acquisition unit102. The difference degree refers to, for example, a degree of difference between the reference video46B1and the virtual viewpoint video46C. The calculation of the difference degree is realized, for example, by comparing the feature amounts obtained by performing pattern matching or Fourier transform.

The difference degree calculation unit104A outputs difference degree information indicating the calculated difference degree to the determination unit104B. The determination unit104B determines whether or not the difference degree indicated by the difference degree information input from the difference degree calculation unit104A is equal to or more than the first threshold value. Examples of a case in which the difference degree is equal to or more than the first threshold value include a case in which the match location between the reference video46B1and the virtual viewpoint video46C is less than 30%.

Note that in the present embodiment, as the first threshold value, a fixed value derived in advance by a sensory test and/or a computer simulation or the like is adopted. The first threshold value does not have to be the fixed value, and may be, for example, a variable value that is changed in response to the instructions received by the reception device52(seeFIG.2) and/or the reception device76.

The determination unit104B compares the difference degree with the first threshold value, and in a case in which the difference degree is equal to or more than the first threshold value, outputs large difference degree information indicating that the difference degree is equal to or more than the first threshold value to the notification processing unit104C. Alternatively, the determination unit104B compares the difference degree with the first threshold value, and in a case in which the difference degree is less than the first threshold value, does not output the large difference degree information and waits for input of the difference degree information from the difference degree calculation unit104A.

When the large difference degree information is input from the determination unit104B, the notification processing unit104C executes a process including a notification process as the specific process described above. Here, the “notification process” refers to a process of giving notification that the difference degree is equal to or more than the first threshold value. A notification destination is, for example, the viewer28. The notification process is realized by transmitting warning indication information to the smart device14by the notification processing unit104C. Here, the warning indication information refers to information for instructing the smart device14to give a warning to the viewer28that the difference degree is equal to or more than the first threshold value. In the example shown inFIG.15, an aspect is described in which the warning indication information is input to the smart device14in a state in which the virtual viewpoint video46C is displayed on the display78of the smart device14.

For example, as shown inFIG.16, in a case in which the warning indication information is input to the smart device14, under the control of the CPU88, a warning message is displayed in a state in which the virtual viewpoint video46C is displayed on the display78and the warning message is parallel to the virtual viewpoint video46C. Note that the display of the warning message on the display78is an example of “visible notification” according to the technology of the present disclosure.

In the example shown inFIG.16, a message “It is quite moved away from the live broadcast video!” is shown as the warning message. Note that the warning message may be a message other than the above, for example, a message such as “Currently, a video completely different from the live broadcast video is being displayed.” or “Since a video completely different from the live broadcast video is being displayed, in a case in which you desires to bring the currently displayed video closer to the live broadcast video, please newly indicate of the viewpoint position and the visual line direction.” may be used. That is, the message need only be the message that allows the viewer28to visually recognize that the difference degree between the video (in the example shown inFIG.16, the virtual viewpoint video46C) that is currently visually recognized by the viewer28via the display78of the smart device14and the reference video46B1(for example, the live broadcast video) displayed on the screen34A of the receiver34is equal to or more than the first threshold value.

In addition, in the example shown inFIG.16, recommendation viewpoint position information and a recommendation visual line direction information are displayed on the display78together with the warning message. The recommendation viewpoint position refers to the viewpoint position42required to reduce the difference degree. The recommendation visual line direction refers to the visual line direction44required to reduce the difference degree. In the example shown inFIG.16, information including an arrow P1indicating the recommendation viewpoint position and a guide message “recommendation viewpoint position” is shown as the recommendation viewpoint position information, and information including an arrow P2indicating the recommendation visual line direction and a guide message “recommendation visual line direction” is shown as the recommendation visual line direction information. The viewer28moves the screen by performing the slide operation, a flick operation, or the like on the touch panel76A to adjust the viewpoint position42.

Note that the CPU88of the smart device14may control the recommendation viewpoint position information such that the arrow P1is shorter as the viewpoint position42approaches the recommendation viewpoint position and the arrow P1is longer as the viewpoint position42is moved away from the recommendation viewpoint position. The CPU88may also change the direction of the arrow P2depending on the difference degree between the current visual line direction44and the recommendation visual line direction.

In addition, here, although the aspect example has been described in which the message is displayed on the display78, the visible notification that the difference degree is equal to or more than the first threshold value may be given by turning on a specific light source (not shown) having a predetermined color (for example, red). In addition, the recommendation viewpoint position and the recommendation visual line direction may be visually recognized by the viewer28by selectively turning on and off the light source having a plurality of colors (for example, a green light source and a yellow light source) and lengthening or shortening a turning-on/off interval.

In addition, examples of a unit that allows the viewer28to perceive the information corresponding to the warning message described above, the recommendation viewpoint position information, and/or the recommendation visual line direction information include a voice reproduction device and/or a vibrator. In this case, for example, the viewer28perceives the information corresponding to the warning message, the recommendation viewpoint position information, and/or the recommendation visual line direction information by audible notification by a voice by using the voice reproduction device and/or tactile notification by using the vibrator. The audible notification by the voice may be audible notification by a bone conduction method. The tactile notification by using the vibrator is realized, for example, by changing a vibration pattern (for example, a vibration cycle and an amplitude) by the vibrator. As described above, the information corresponding to the warning message described above, the recommendation viewpoint position information, and/or the recommendation visual line direction information need only be perceived by the viewer28by the visible notification by the display78or the like, the audible notification by using the voice reproduction device, and/or the tactile notification by the vibrator or the like.

The determination unit104B compares the difference degree with the first threshold value, and in a case in which the difference degree is less than the first threshold value, does not output the large difference degree information. Therefore, as shown inFIG.17, the virtual viewpoint video46C is displayed on the display78of the smart device14under the control of the CPU88, but the warning message, the recommendation viewpoint position information, and the recommendation visual line direction information are not displayed.

For example, as shown inFIG.18, the setting unit108includes a difference degree change amount calculation unit108A, a change amount determination unit108B, and a mode indication unit108C. The setting unit108selectively sets a following mode and a non-following mode. The following mode refers to an operation mode in which the execution unit104executes the notification process while following the change in the virtual viewpoint video46C, and the non-following mode refers to an operation mode in which the execution unit104does not execute the notification process while following the change in the virtual viewpoint video46C.

The difference degree change amount calculation unit108A acquires the difference degree information from the difference degree calculation unit104A. That is, the difference degree information is acquired in synchronization with an acquisition timing of the difference degree information by the determination unit104B (seeFIG.15), and the change amount of the difference degree is calculated based on the acquired difference degree information. Specifically, the difference degree change amount calculation unit108A first acquires the difference degree information at different timings. Here, in a case in which N is an integer of 1 or more, the different timings refer to, for example, an Nth acquisition timing and an N+1th acquisition timing. Next, the difference degree change amount calculation unit108A calculates the change amount of the difference degree indicated by the difference degree information acquired at the Nth acquisition timing to the difference degree indicated by the difference degree information acquired at the N+1th acquisition timing. Here, examples of the “change amount” include an absolute value of the difference between the difference degree indicated by the difference degree information acquired at the Nth acquisition timing and the difference degree indicated by the difference degree information acquired at the N+1th acquisition timing. Note that instead of the absolute value of the difference, a ratio of the difference degree indicated by the difference degree information acquired at the N+1th acquisition timing to the difference degree indicated by the difference degree information acquired at the Nth acquisition timing may be used, or any information may be used as long as the information indicates the change amount of the difference degree.

The change amount determination unit108B determines whether or not the change amount calculated by the difference degree change amount calculation unit108A is less than a second threshold value. Note that in the present embodiment, as the second threshold value, a fixed value derived in advance by a sensory test and/or a computer simulation or the like is adopted. The second threshold value does not have to be the fixed value, and may be, for example, a variable value that is changed in response to the instructions received by the reception device52(seeFIG.2) and/or the reception device76.

The change amount determination unit108B outputs the information indicating that the change amount is less than the second threshold value to the mode indication unit108C in a case in which the change amount calculated by the difference degree change amount calculation unit108A is less than the second threshold value. In addition, the change amount determination unit108B outputs the information indicating that the change amount is equal to or more than the second threshold value to the mode indication unit108C in a case in which the change amount calculated by the difference degree change amount calculation unit108A is equal to or more than the second threshold value.

The mode indication unit108C outputs following mode indication information for indicating the following mode as the operation mode of the execution unit104in a case in which the information indicating that the change amount is less than the second threshold value is input from the change amount determination unit108B. The mode indication unit108C outputs non-following mode indication information for indicating the non-following mode as the operation mode of the execution unit104in a case in which the information indicating that the change amount is equal to or more than the second threshold value is input from the change amount determination unit108B. In a case in which the following mode indication information is input from the mode indication unit108C, the execution unit104is operated in the following mode as indicated by the following mode indication information. In a case in which the non-following mode indication information is input from the mode indication unit108C, the execution unit104is operated in the non-following mode as indicated by the non-following mode indication information.

In this way, the setting unit108sets the following mode as the operation mode of the execution unit104in a case in which the change amount of the difference degree is less than the second threshold value, and sets the non-following mode as the operation mode of the execution unit104in a case in which the change amount of the difference degree is equal to or more than the second threshold value.

Next, an operation of the information processing system10will be described.

First, an example of a flow of the display control process executed by the CPU58of the information processing apparatus12will be described with reference toFIG.19. Note that, here, for convenience of description, the description will be made on the premise that the reference video46B1is already generated by the reference video generation unit100A and the virtual viewpoint video46C is already generated by the virtual viewpoint video generation unit100B.

In the display control process shown inFIG.19, first, in step ST200, the acquisition unit102acquires the reference video46B1generated by the reference video generation unit100A from the reference video generation unit100A, and then the display control process proceeds to step ST202.

In step ST202, the reference video control unit106A displays the reference video46B1on the receiver34by transmitting the reference video46B1acquired in step ST200to the receiver34. That is, the receiver34receives the reference video46B1, and displays the received reference video46B1on the screen34A.

In subsequent step ST204, the acquisition unit102acquires the virtual viewpoint video46C generated by the virtual viewpoint video generation unit100B from the virtual viewpoint video generation unit100B, and then the display control process proceeds to step ST206.

In step ST206, the virtual viewpoint video control unit106B displays the virtual viewpoint video46C on the smart device14by transmitting the virtual viewpoint video46C acquired in step ST204to the smart device14. That is, the smart device14receives the virtual viewpoint video46C, and displays the received virtual viewpoint video46C on the display78.

In subsequent step ST208, the control unit106determines whether or not a condition for terminating the display control process (hereinafter, referred to as a “display control process termination condition”) is satisfied. Examples of the display control process termination condition include a condition that an instruction for terminating the display control process is received by the reception device52or76. In a case in which the display control process termination condition is not satisfied in step ST208, a negative determination is made, and the display control process proceeds to step ST200. In a case in which the display control process termination condition is satisfied in step ST208, a positive determination is made, and the display control process is terminated.

Next, an example of a flow of the setting process executed by the CPU58of the information processing apparatus12will be described with reference toFIG.20. Note that, here, for convenience of description, the description will be made on the premise that the difference degree is calculated by the difference degree calculation unit104A, and the calculated difference degree information is output in synchronization with the determination unit104B and the difference degree change amount calculation unit108A.

In the setting process shown inFIG.20, first, in step ST250, the difference degree change amount calculation unit108A acquires the difference degree information from the difference degree calculation unit104A at different timings, and then the setting process proceeds to step ST252.

In step ST252, the difference degree change amount calculation unit108A calculates the change amount for each difference degree indicated by each difference degree information acquired at different timings in step ST250, and then the setting process proceeds to step ST254.

In step ST254, the change amount determination unit108B determines whether or not the change amount calculated in step ST252is less than the second threshold value. In step ST254, in a case in which the change amount calculated in step ST252is equal to or more than the second threshold value, a negative determination is made, and the setting process proceeds to step ST256. In step ST254, in a case in which the change amount calculated in step ST252is less than the second threshold value, a positive determination is made, and the setting process proceeds to step ST258.

In step ST256, the mode indication unit108C outputs the non-following mode indication information to the execution unit104, and then the setting process proceeds to step ST260. By executing the process of step ST256, the execution unit104is operated in the non-following mode as indicated by the non-following mode indication information input from the mode indication unit108C.

In step ST258, the mode indication unit108C outputs the following mode indication information to the execution unit104, and then the setting process proceeds to step ST260. By executing the process of step ST258, the execution unit104is operated in the following mode as indicated by the following mode indication information input from the mode indication unit108C.

In subsequent step ST260, the setting unit108determines whether or not a condition for terminating the setting process (hereinafter, referred to as a “setting process termination condition”) is satisfied. Examples of the setting process termination condition include a condition that the reception device52or76receives an instruction for terminating the setting process. In a case in which the setting process termination condition is not satisfied in step ST260, a negative determination is made, and the setting process proceeds to step ST250. In a case in which the setting process termination condition is satisfied in step ST260, a positive determination is made, and the setting process is terminated.

Next, an example of a flow of the information processing executed by the CPU58of the information processing apparatus12will be described with reference toFIG.21. Note that, here, for convenience of description, the description will be made on the premise that the reference video46B1is already generated by the reference video generation unit100A and the virtual viewpoint video46C is already generated by the virtual viewpoint video generation unit100B. In addition, here, the description will be made on the premise that the virtual viewpoint video46C generated by the virtual viewpoint video generation unit100B is displayed on the display78of the smart device14. Further, here, the description will be made on the premise that the following mode or the non-following mode is set as the operation mode for the execution unit104.

In the information processing shown inFIG.21, first, in step ST300, the acquisition unit102acquires the reference video46B1from the reference video generation unit100A, and then the information processing proceeds to step ST302.

In step ST302, the acquisition unit102acquires the virtual viewpoint video46C from the virtual viewpoint video generation unit100B, and then the information processing proceeds to step ST304.

In step ST304, the difference degree calculation unit104A calculates the difference degree between the reference video46B1acquired in step ST300and the virtual viewpoint video46C acquired in step ST302, and then the information processing proceeds to step ST306.

In step ST306, the determination unit104B determines whether or not the operation mode of the execution unit104is the following mode. In step ST306, in a case in which the operation mode of the execution unit104is the non-following mode, a negative determination is made, and the information processing proceeds to step ST312. In step ST306, in a case in which the operation mode of the execution unit104is the following mode, a positive determination is made, and the information processing proceeds to step ST308.

In step ST308, the determination unit104B determines whether or not the difference degree calculated in step ST304is equal to or more than the first threshold value. In a case in which the difference degree is less than the first threshold value in step ST308, a negative determination is made, and the information processing proceeds to step ST312. In a case in which the difference degree is equal to or more than the first threshold value in step ST308, a positive determination is made, and the information processing proceeds to step ST310.

In step ST310, the notification processing unit104C executes the notification process, and then the information processing proceeds to step ST312. By executing the notification process by the notification processing unit104C, warning information is transmitted to the smart device14by the notification processing unit104C. In a case in which the warning information is received, the smart device14displays the warning message, the recommendation viewpoint position information, and the recommendation visual line direction information on the display78together with the virtual viewpoint video46C. In this way, the visible notification of the warning message, the recommendation viewpoint position information, and the recommendation visual line direction information is given to notify the viewer28that the difference degree is equal to or more than the first threshold value.

In subsequent step ST312, the execution unit104determines whether or not a condition for terminating the information processing (hereinafter, referred to as an “information processing termination condition”) is satisfied. Examples of the information processing termination condition include a condition that the reception device52or76receives an instruction for terminating the information processing. In a case in which the information processing termination condition is not satisfied in step ST312, a negative determination is made, and the information processing proceeds to step ST300. In a case in which the information processing termination condition is satisfied in step ST312, a positive determination is made, and the information processing is terminated.

As described above, in the information processing system10, the acquisition unit102acquires the virtual viewpoint video46C and the reference video46B1, and the execution unit104executes the specific process that contributes to setting of the difference degree to be less than the first threshold value in a case in which the difference degree between the virtual viewpoint video46C and the reference video46B1is equal to or more than the first threshold value. Therefore, it is possible to contribute to the acquisition of the virtual viewpoint video46C in which the difference degree from the reference video46B1is less than the first threshold value.

Note that, in the above, the virtual viewpoint video46C has been described as an example, but instead of the virtual viewpoint video46C, the imaging video46B itself may be used as a time slice free viewpoint video, or the virtual viewpoint video46C and the time slice free viewpoint video may be used in combination. In addition, not limited to the moving image, the free viewpoint video of the still image may be used. As a result, it is possible to contribute to the acquisition of the free viewpoint video in which the difference degree from the reference video46B1is less than the first threshold value.

In addition, in the information processing system10, the execution unit104executes the notification process as the specific process. By executing the notification process, the viewer28is notified of the warning message indicating that the difference degree is equal to or more than the first threshold value. Therefore, the viewer28can be made aware that the difference degree between the virtual viewpoint video46C and the reference video46B1is equal to or more than the first threshold value.

In addition, the information processing system10executes the notification process in which the viewer28is notified of the warning message by performing the visible notification, the audible notification, and/or the tactile notification. Therefore, the viewer28can be made perceive that the difference degree between the virtual viewpoint video46C and the reference video46B1is equal to or more than the first threshold value.

In addition, in the information processing system10, the execution unit104executes the specific process depending on the change in the virtual viewpoint video46C. Therefore, even in a case in which the virtual viewpoint video46C is changed, it can immediately contribute to the acquisition of the virtual viewpoint video of which the difference degree with the reference video46B1is less than the first threshold value.

In addition, in the information processing system10, the following mode and the non-following mode are selectively set by the setting unit108. Therefore, it is possible to reduce the power consumption as compared to a case in which the specific process is always executed while following the change in the virtual viewpoint video46C.

In addition, in the information processing system10, the setting unit108sets the following mode in a case in which the change amount of the difference degree is less than the second threshold value, and sets the non-following mode in a case in which the change amount of the difference degree is equal to or more than the second threshold value. Therefore, the specific process can be prevented from being executed contrary to the intention of the viewer28.

In addition, in the information processing system10, the control unit106displays the virtual viewpoint video46C on the display78of the smart device14. Therefore, it is possible for the viewer28to visually recognize the virtual viewpoint video46C.

In addition, in the information processing system10, the control unit106displays the reference video46B1on the screen34A of the receiver34. Therefore, it is possible for the viewer of the receiver34to visually recognize the reference video46B1.

In addition, in the information processing system10, the live broadcast video is adopted as the reference video46B1. Therefore, it is possible to contribute to the acquisition of the virtual viewpoint video46C in which the difference degree from the live broadcast video is less than the first threshold value.

In addition, in the information processing system10, the video obtained by imaging the imaging region by the reference imaging apparatus is used as the reference video46B1. Therefore, as compared to a case in which a video other than the video obtained by imaging the imaging region (for example, a virtual video) is used as the reference video, it is possible for the viewer28or the like to easily grasp the difference between the video obtained by imaging and the virtual viewpoint video46C.

In addition, in the information processing system10, the viewpoint position42and the visual line direction44as indicated from the outside are used. That is, the viewpoint position42and the visual line direction44are decided in response to the viewpoint visual line indication received by the touch panel76A. Therefore, it is possible to generate the virtual viewpoint video46C in a case in which the imaging region is observed from the viewpoint position and the visual line direction intended by the viewer28.

Further, in the information processing system10, as the specific process, the process that contributes to setting the difference degree to be less than the first threshold value is adopted. Therefore, it is possible to contribute to the acquisition of the virtual viewpoint video46C in which the difference degree from the reference video46B1is less than the first threshold value.

Note that in the embodiment described above, the aspect example has been described in which the reference video46B1and the virtual viewpoint video46C are generated by the information processing apparatus12, but the technology of the present disclosure is not limited to this. The reference video46B1and/or the virtual viewpoint video46C may be generated by a device different from the information processing apparatus12(hereinafter, referred to as a “video generation device”), and the video generated by the video generation device may be acquired by the acquisition unit102.

In the embodiment described above, the notification process has been described as an example of the specific process executed by the execution unit104, but the technology of the present disclosure is not limited to this, and the information processing shown inFIG.22may be executed instead of the information processing (seeFIG.21) described in the embodiment. The information processing shown inFIG.22is different from the information processing shown inFIG.21in that step ST310A is provided instead of step ST310. In step ST310A, a limitation processing unit104D (seeFIG.23) executes a limitation process.

The limitation process is a process for limiting a range in which the viewpoint position42can be indicated to a viewpoint position range R1(seeFIG.24) determined as a range in which the difference degree is set to be less than the first threshold value and limiting a range in which the visual line direction44can be indicated to a visual line direction range R2(seeFIG.24) determined as the range in which the difference degree is set to be less than the first threshold value.

For example, as shown inFIG.23, the limitation processing unit104D outputs viewpoint visual line limitation instruction information to the virtual viewpoint video generation unit100B by executing the limitation process. The viewpoint visual line limitation instruction information is information including viewpoint position limitation instruction information and visual line direction limitation instruction information. The viewpoint position limitation instruction information refers to information indicating an instruction for limiting the range in which the viewpoint position42can be indicated to the viewpoint position range R1(seeFIG.24), and the visual line direction limitation instruction information refers to information indicating an instruction for limiting the range in which the visual line direction44can be indicated to the visual line direction range R2(seeFIG.24).

In this case, for example, as shown inFIG.24, the viewpoint position42is forcibly set in the viewpoint position range R1and the visual line direction44is forcibly set in the visual line direction range R2. Stated another way, the viewpoint position42follows the viewpoint position range R1, and the visual line direction44follows the visual line direction range R2. For example, as shown by a two dot chain line inFIG.24, even in a case in which the viewpoint position42and the visual line direction44are once moved away from the viewpoint position range and the visual line direction range, the viewpoint position42and the visual line direction44are returned again in the viewpoint position range and the visual line direction range.

The disposition position of the viewpoint position42in the viewpoint position range R1is decided depending on a positional relationship between the viewpoint position range R1and the current viewpoint position42. For example, in the viewpoint position range R1, the viewpoint position42is changed to a position closest to the current viewpoint position42. In addition, the direction of the visual line direction in the visual line direction range R2is decided depending on a positional relationship between the visual line direction range R2and the current visual line direction44. For example, in the visual line direction range R2, the visual line direction44is changed to a direction closest to the current visual line direction44.

For example, as shown inFIG.25, the virtual viewpoint video generation unit100B includes a viewpoint visual line range determination unit100B1and a video generation execution unit100B2. The viewpoint visual line range determination unit100B1acquires the viewpoint visual line indication from the smart device14, and also acquires the viewpoint visual line limitation instruction information from the limitation processing unit104D.

The viewpoint visual line range determination unit100B1determines whether or not the viewpoint position42is in the viewpoint position range R1and the visual line direction44is in the visual line direction range R2based on the viewpoint visual line indication and the viewpoint visual line limitation indication information. In a case in which the viewpoint position42is outside the viewpoint position range R1and the visual line direction44is outside the visual line direction range R2, the viewpoint visual line range determination unit100B1adjusts the viewpoint visual line indication such that the current viewpoint position42is set in the viewpoint position range R1and the current visual line direction44is set in the visual line direction range R2. Moreover, the viewpoint visual line range determination unit100B1outputs the adjusted viewpoint visual line indication to the video generation execution unit100B2. The video generation execution unit100B2generates the virtual viewpoint video46C in response to the viewpoint visual line indication input from the viewpoint visual line range determination unit100B1. As a result, the virtual viewpoint video46C of which the difference degree with the reference video46B1is less than the first threshold value is generated depending on the viewpoint position42in the viewpoint position range R1and the visual line direction44in the visual line direction range R2. Therefore, it is possible to maintain the difference degree between the reference video46B1and the virtual viewpoint video46C to be less than the first threshold value.

In addition, for example, as shown inFIG.26, the limitation processing unit104D may output the viewpoint visual line limitation instruction information to the smart device14in addition to the viewpoint visual line range determination unit100B1. In this case, for example, as shown inFIG.27, in the smart device14, under the control of the CPU88, a rectangular frame R1A indicating an outer edge of the viewpoint position range R1superimposed on the virtual viewpoint video46C is displayed the display78and a guide message “Please indicate the viewpoint position in the rectangular frame.” is displayed. For example, as shown inFIG.28, in a case in which the viewpoint position indication from the viewer28is received by the touch panel76A in the rectangular frame R1A, the viewpoint position42is changed to a position in the viewpoint position range R1.

In addition, in a case in which the viewpoint position indication is received by the touch panel76A, the rectangular frame R1A is deleted from the display78. Moreover, subsequently, for example, as shown inFIG.29, in the smart device14, under the control of the CPU88, a circular frame R2A indicating an outer edge of the visual line direction range R2superimposed on the virtual viewpoint video46C is displayed on the display78and a guide message “Please indicate the visual line direction toward an inside of the circular frame.” is displayed. In this case, for example, in a case in which the visual line direction indication (for example, the slide operation) from the viewer28is received by the touch panel76A toward the circular frame R2A, the visual line direction44is changed to a direction in the visual line direction range R2.

In this way, in a case in which the viewpoint position42is changed to the position in the viewpoint position range R1and the visual line direction44is changed to the direction in the visual line direction range R2, the virtual viewpoint video46C is generated by the virtual viewpoint video generation unit100B depending on the changed viewpoint position42and the changed visual line direction44.

In addition, in the example shown inFIG.23, an aspect example is described in which the limitation process is executed by the limitation processing unit104D, but the technology of the present disclosure is not limited to this. For example, as shown inFIG.30, the execution unit104may include a change processing unit104E instead of the limitation processing unit104D. In the example shown inFIG.23, the limitation processing unit104D is not shown, but the limitation processing unit104D can be present together with the change processing unit104E.

The change processing unit104E executes a change process. The change process refers to a process of performing, on the virtual viewpoint video46C, change for setting the difference degree to be less than the first threshold value. That is, in a case in which the change processing unit104E executes the change process, the virtual viewpoint video46C is changed such that the difference degree is set to be less than the first threshold value. In a case in which the large difference degree information is input from the determination unit104B, the change processing unit104E generates changeable amount information based on the difference degree information input from the difference degree calculation unit104A via the determination unit104B, and outputs the generated changeable amount information to the virtual viewpoint video generation unit100B. The changeable amount information refers to a limit value of the change amount required to change the virtual viewpoint video46C, that is, an upper limit value of the change amount required to set the difference degree indicated by the difference degree information to be less than the first threshold value (hereinafter, referred to as a “changeable amount”).

For example, as shown inFIG.31, the virtual viewpoint video generation unit100B includes the video generation execution unit100B2and a viewpoint visual line indication adjustment unit100B3. The viewpoint visual line indication adjustment unit100B3acquires the viewpoint visual line indication from the smart device14and acquires the changeable amount information from the change processing unit104E.

The viewpoint visual line indication adjustment unit100B3changes the viewpoint position42and the visual line direction44by adjusting the viewpoint visual line indication. Specifically, the viewpoint visual line indication adjustment unit100B3adjusts the viewpoint visual line indication in the changeable amount indicated by the changeable amount information. Here, the adjustment of the viewpoint visual line indication refers to the adjustment of the viewpoint position indication and the visual line direction indication. The viewpoint visual line indication adjustment unit100B3outputs the adjusted viewpoint visual line indication to the video generation execution unit100B2. The video generation execution unit100B2generates the virtual viewpoint video46C by using the plurality of imaging videos in response to the viewpoint visual line indication input from the viewpoint visual line indication adjustment unit100B3.

As a result, the virtual viewpoint video46C is changed by the change amount required to set the difference degree to be less than the first threshold value, so that the difference degree between the virtual viewpoint video46C and the reference video46B1can be maintained to be less than the first threshold value. In addition, since the viewpoint position indication and the visual line direction indication are adjusted, it is possible to easily change the virtual viewpoint video46C of which the difference degree with the reference video46B1is set to be less than the first threshold value as compared to a case in which neither the viewpoint position indication nor the visual line direction indication is adjusted.

Note that, here, although the aspect example has been described in which both the viewpoint position indication and the visual line direction indication are adjusted, the technology of the present disclosure is not limited to this, and any one of the viewpoint position indication or the visual line direction indication may be adjusted by the viewpoint visual line indication adjustment unit100B3such that the difference degree is set to be less than the first threshold value.

In addition, in the example shown inFIG.18, the change amount determination unit108B has been described as an example, but the technology of the present disclosure is not limited to this. For example, as shown inFIG.32, the setting unit108may include a change frequency determination unit108B1instead of the change amount determination unit108B. The change frequency determination unit108B1determines whether or not a frequency at which the difference degree changes is less than a third threshold value. Here, examples of the “frequency at which the difference degree changes” include a frequency at which the difference degree exceeds the first threshold value described (for example, the frequency per unit time). The change frequency determination unit108B1outputs information indicating that the frequency is less than the third threshold value to the mode indication unit108C in a case in which the frequency at which the difference degree changes is less than the third threshold value, and outputs information indicating that the frequency is equal to or more than the third threshold value to the mode indication unit108C in a case in which the frequency at which the difference degree changes is equal to or more than the third threshold value. As a result, as in the embodiment described above, the execution unit104is operated in the following mode in a case in which the frequency at which the difference degree changes is less than the third threshold value, and is operated in the non-following mode in a case in which the frequency at which the difference degree changes is equal to or more than the third threshold value. Therefore, it is possible to reduce the discomfort given to the viewer28due to the frequent execution of the specific process as compared to a case in which the specific process is executed by the execution unit104regardless of the frequency at which the difference degree between the virtual viewpoint video46C and the reference video46B1changes.

In addition, in the embodiment described above, the aspect example has been described in which the specific process is executed by the execution unit104regardless of the posture of the smart device14, but the technology of the present disclosure is not limited to this, and the specific process may be executed or not executed depending on the posture of the smart device14. In this case, based on the angle information measured by the gyro sensor74, the CPU88of the smart device14detects the posture of the smart device14. Note that, here, the CPU88is an example of a “detection unit (detector)” according to the technology of the present disclosure. For example, as shown inFIG.33, the CPU88outputs posture information indicating the posture of the smart device14to the execution unit104. The execution unit104includes a posture determination unit104F, and the posture determination unit104F determines whether or not the posture indicated by the posture information input from the CPU88is a predetermined posture. The predetermined posture refers to a posture in which the display78is not vertically downward, for example.

In a case in which the posture indicated by the posture information is the predetermined posture, the posture determination unit104F outputs execution instruction information for giving an instruction of the execution of the specific process to the difference degree calculation unit104A, the determination unit104B, and the notification processing unit104C. In a case in which the execution instruction information is input to the difference degree calculation unit104A, the determination unit104B, and the notification processing unit104C, the difference degree calculation unit104A, the determination unit104B, and the notification processing unit104C are operated to execute the specific process described above. In addition, in a case in which the posture indicated by the posture information is not the predetermined posture, the posture determination unit104F outputs non-execution instruction information for giving an instruction of the non-execution of the specific process to the difference degree calculation unit104A, the determination unit104B, and the notification processing unit104C. In a case in which the non-execution instruction information is input to the difference degree calculation unit104A, the determination unit104B, and the notification processing unit104C, the difference degree calculation unit104A, the determination unit104B, and the notification processing unit104C are not operated, and the specific process described above is not executed. Therefore it is possible to execute or not execute the specific process by changing the posture of the smart device14.

In addition, although the fixed value is adopted as the first threshold value described in the embodiment described above, the technology of the present disclosure is not limited to this, and the variable value which is changed in response to the instructions received by the reception device52(seeFIG.2) and/or the reception device76may be adopted. In this case, for example, as shown inFIG.34, in a case in which first threshold value change instruction information for giving an instruction for changing the first threshold value is received by the touch panel76A of the smart device14, the change of the first threshold value may be executed by the execution unit104according to the first threshold value change instruction information. Here, the first threshold value change instruction information is an example of “instruction information” according to the technology of the present disclosure.

In the example shown inFIG.34, the execution unit104includes an instruction information reception unit104G, and the first threshold value change instruction information is transmitted from the smart device14to the instruction information reception unit104G in a case in which the first threshold value change instruction information is received by the touch panel76A of the smart device14. The first threshold value change instruction information includes information indicating the first threshold value or information indicating a degree of the change in the first threshold value. The instruction information reception unit104G receives the first threshold value change instruction information to output the received first threshold value change instruction information to the determination unit104B. In a case in which the first threshold value change instruction information is input to the determination unit104B, the determination unit104B changes the first threshold value used for comparison with the difference degree according to the first threshold value change instruction information. Therefore, as compared to a case in which the first threshold value is always fixed, it is possible to easily determine the intention of the viewer28or the like for the change in the first threshold value.

In addition, in the embodiment described above, the aspect example has been described in which the display control process (seeFIG.19), the setting process (seeFIG.20), and the information processing (seeFIGS.21and22) are executed by the CPU58of the information processing apparatus12, but the technology of the present disclosure is not limited to this. At least one of the display control process, the setting process, or the information processing may be executed by the CPU88of the smart device14. In the following, in a case in which a distinction is not necessary, the display control process, the setting process, and the information processing are referred to as an “information processing apparatus side process”.

FIG.35shows a configuration example of the smart device14in a case in which the information processing apparatus side process is executed by the CPU88of the smart device14. For example, as shown inFIG.35, the storage90stores the information processing apparatus side program. In addition, the CPU88executes the information processing by being operated as the acquisition unit102and the execution unit104according to the information processing program60A. The CPU88executes the display control process by being operated as the control unit106according to the display control program60B. Further, the CPU88executes the setting process by being operated as the setting unit108according to the setting program60C.

Note that in the example shown inFIG.35, the smart device14is an example of an “information processing apparatus” according to the technology of the present disclosure. Note that in the example shown inFIG.35, the smart device14is described as an example of the “information processing apparatus” according to the technology of the present disclosure, instead of the smart device14, various devices equipped with an arithmetic device, such as a head up display, a head mounted display, a personal computer and/or a wearable terminal, can also be adopted as the “information processing apparatus” according to the technology of the present disclosure.

In addition, in the embodiment described above, the soccer stadium22has been described as an example, but it is merely an example, and any place, such as a baseball stadium, a rugby stadium, a curling stadium, an athletics stadium, a swimming pool, a concert hall, an outdoor music hall, and a theater venue, may be adopted as long as the plurality of imaging apparatuses and the plurality of sound collection devices100can be installed.

In addition, in the embodiment described above, the wireless communication method using the base station20has been described as an example, but it is merely an example, and the technology of the present disclosure is established even in the wired communication method using the cable.

In addition, in the embodiment described above, the unmanned aerial vehicle27has been described as an example, but the technology of the present disclosure is not limited to this, and the imaging region may be imaged by the imaging apparatus18suspended by a wire (for example, a self-propelled imaging apparatus that can move along the wire).

In addition, in the embodiment described above, the computers50and70have been described as an example, but the technology of the present disclosure is not limited to theses. For example, instead of the computers50and/or70, a device including an ASIC, an FPGA, and/or a PLD may be applied. In addition, instead of the computers50and/or70, a combination of a hardware configuration and a software configuration may be used.

In addition, in the embodiment described above, the information processing apparatus side program is stored in the storage60, but the technology of the present disclosure is not limited to this, and as shown inFIG.36, for example, the information processing apparatus side program may be stored in any portable storage medium500, such as an SSD or a USB memory, which is a non-transitory storage medium. In this case, the information processing apparatus side program stored in the storage medium500is installed in the computer50, and the CPU58executes the information processing apparatus side process according to the information processing apparatus side program.

In addition, the information processing apparatus side program may be stored in a storage unit of another computer or a server device connected to the computer50via a communication network (not shown), and the information processing apparatus side program may be downloaded to the information processing apparatus12in response to the request of the information processing apparatus12. In this case, the information processing apparatus side process based on the downloaded information processing apparatus side program is executed by the CPU58of the computer50.

In addition, in the embodiment described above, the CPU58has been described as an example, but the technology of the present disclosure is not limited to this, and a GPU may be adopted. In addition, a plurality of CPUs may be adopted instead of the CPU58. That is, the information processing apparatus side process may be executed by one processor or a plurality of physically separated processors. In addition, instead of the CPU88, a GPU may be adopted, a plurality of CPUs may be adopted, or one processor or a plurality of physically separated processors may be adopted to execute various processes.

The following various processors can be used as a hardware resource for executing the information processing apparatus side process. Examples of the processor include a CPU, which is a general-purpose processor that functions as software, that is, the hardware resource for executing the information processing apparatus side process according to the program, as described above. In addition, another example of the processor includes a dedicated electric circuit which is a processor having a circuit configuration specially designed for executing a specific process, such as an FPGA, a PLD, or an ASIC. A memory is also built in or connected to each processor, and each processor executes the information processing apparatus side process by using the memory.

The hardware resource for executing the information processing apparatus side process may be configured by one of the various processors, or may be a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). In addition, the hardware resource for executing the information processing apparatus side process may be one processor.

As an example of configuring the hardware resource with one processor, first, as represented by a computer such as a client computer or a server, there is a form in which one processor is configured by a combination of one or more CPUs and software, and the processor functions as the hardware resource for executing the information processing apparatus side process. Secondly, as represented by SoC, there is an aspect in which a processor that realizes the functions of the whole system including a plurality of the hardware resources for executing the information processing apparatus side process with one IC chip is used. In this way, the information processing apparatus side process is realized by using one or more of the various processors described above as the hardware resource.

Further, as the hardware structure of these various processors, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined can be used.

In addition, the information processing apparatus side process described above is merely an example. Therefore, it is needless to say that unnecessary steps may be deleted, new steps may be added, or the process order may be changed within a range that does not deviate from the gist.

The contents described and shown above are the detailed description of the parts according to the technology of the present disclosure, and are merely examples of the technology of the present disclosure. For example, the description of the configuration, the function, the action, and the effect above are the description of examples of the configuration, the function, the action, and the effect of the parts according to the technology of the present disclosure. Accordingly, it is needless to say that unnecessary parts may be deleted, new elements may be added, or replacements may be made with respect to the contents described and shown above within a range that does not deviate from the gist of the technology of the present disclosure. In addition, in order to avoid complications and facilitate understanding of the parts according to the technology of the present disclosure, in the contents described and shown above, the description of common technical knowledge and the like that do not particularly require description for enabling the implementation of the technology of the present disclosure are omitted.

In the present specification, “A and/or B” is synonymous with “at least one of A or B”. That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. In addition, in the present specification, in a case in which three or more matters are associated and expressed by “and/or”, the same concept as “A and/or B” is applied.

All of the documents, the patent applications, and the technical standards described in the present specification are incorporated in the present specification by referring to the same extent as a case in which individual document, patent application, and technical standard are specifically and individually noted to be incorporated by reference.

Regarding the embodiment described above, the following supplementary note will be further disclosed.

(Supplementary Note 1)

An information processing apparatus including a processor, and a memory built in or connected to the processor,in which the processor acquires a free viewpoint video based on at least one image among a plurality of images obtained by imaging an imaging region by a plurality of imaging apparatuses, and a reference video, the free viewpoint video indicating the imaging region in a case in which the imaging region is observed from a specific viewpoint position and a specific visual line direction, andexecutes a specific process in a case in which a difference degree between the acquired free viewpoint video and the acquired reference video is equal to or more than a first threshold value.