Method and apparatus for displaying light field video data

A method and an apparatus for displaying light field video data are described. A receiving unit of the apparatus receives light field video data including a plurality of light field video sequences. Each light field video frames of of the light field video sequences has multiple selectable viewpoints. A light field video frame is provided by an operation unit and displayed on a display device with a first viewpoint. An input unit receives an input indicating a viewpoint different from the first viewpoint. The displayed light field video frame is then displayed with the indicated viewpoint. In one embodiment, two light field video frames of two light field video sequences are displayed respectively in main and side windows on the display device.

TECHNICAL FIELD

The present invention relates to a method and an apparatus for displaying light field video data, particularly with different viewpoints of the light field video data. The invention also relates to a computer readable medium suitable for such a method and an apparatus.

BACKGROUND

Light field capture data, which can be captured for example by a plenoptic camera or an array of cameras, allows refocusing of the capture after the snapshot, estimating the depth of the scene in the capture, and many other flexible editing options for the capture data. In order to visualize such light field content, several user interfaces have been proposed to display and enable an operator to review and edit individual images of the light field capture [I].

Besides individual light field images, there are already devices able to capture light field videos, for example, the Raytrix plenoptic camera. It can be expected that more light field videos will be available and used in the near future. However, the existing user interfaces are only applicable to the visualization of light field images and are not designed to visualize light field videos. There is a need for a solution that enables users to review and make use of light field videos.

SUMMARY

Therefore, it is an objective to propose solutions for displaying and visualizing light field video data. The proposed method and system provide solutions for displaying light field video data, as well as easy and intuitive navigation for reviewing the same.

According to one embodiment, a method for displaying light field video data is proposed. The light field video data includes a plurality of light field video sequences. Each light field video frame of the light field video sequences has multiple selectable viewpoints. The method comprises: receiving the light field video data including a plurality of light field video sequences; displaying a first light field video frame of a first light field video sequence with a first viewpoint; receiving an input indicating a viewpoint of the first light field video frame different from the first viewpoint; and in response to the indicating input, displaying the first light field video frame with the indicated viewpoint.

In one embodiment, the light field video frames of the first light field video sequence are temporally sequenced, and the method comprises: in response to the indicating input, displaying the light field video frames of the first light field video sequence temporally subsequent to the first light field video frame with the indicated viewpoint.

In one embodiment, the received input indicating the viewpoint is generated by selecting a region of the displayed first light field video frame corresponding to the indicated viewpoint.

In one embodiment, the first light field video frame of the first light field video sequence is displayed in a main window on a screen. The method further comprises displaying a second light field video frame of a second light field video sequence with a second viewpoint in a side window on the screen.

In one embodiment, the first and second displayed light field video frames are synchronized temporally or by other selectable characteristics derived from the contents of the light field video data.

In one embodiment, the method further comprises: receiving an input selecting the displayed second light field video frame with the second viewpoint in the side window; and, in response to the selecting input, (a) displaying the second light field video frame with the second viewpoint in the main window, replacing the first light field video frame, when the second viewpoint is in close proximity to the first viewpoint, or (b) displaying the first light field video frame in the main window with a changing viewpoint, producing a smooth transition over time between the first and second viewpoints, when the second viewpoint is not in close proximity to the first viewpoint. Smoothing can either be linear or done according to some user-defined curve, spline, etc. that will slow down or accelerate the transition from first viewpoint to the second viewpoint over time.

According to one embodiment, an apparatus configured to provide light field video data to a display device is introduced. The apparatus comprises a receiving unit, an operation unit, an input unit, and optionally a storage unit. The receiving unit is configured to receive light field video data including a plurality of light field video sequences. Each light field video frame of the light field video sequences has multiple selectable viewpoints. The storage unit is configured to store the received light field video data. The operation unit is configured to provide a first light field video frame of a first light field video sequence with a first viewpoint to a display device. The input unit is configured to receive an input indicating a viewpoint of the first light field video frame different from the first viewpoint. The operation unit is configured to provide the first light field video frame with the indicated viewpoint to the display device in response to the indicating input.

In one embodiment, the light field video frames of the first light field video sequence are temporally sequenced, and the operation unit is configured to provide the light field video frames of the first light field video sequence temporally subsequent to the first light field video frame to the display device with the indicated viewpoint in response to the indicating input.

In one embodiment, the operation unit is configured to provide a second light field video frame of a second light field video sequence with a second viewpoint to the display device. Preferably, the operation unit is configured to temporally synchronize the first and second light field video frames.

In one embodiment, the input unit is configured to receive an input selecting the second light field video frame with the second viewpoint. The operation unit is configured to, in response to the selecting input, provide to the display device (a) the second light field video frame with the second viewpoint when the second viewpoint is in close proximity to the first viewpoint, or (b) the first light field video frame with a changing viewpoint, producing a smooth transition between the first and second viewpoints, when the second viewpoint is not in close proximity to the first viewpoint.

Also, a computer readable storage medium has stored therein instructions enabling displaying light field video data, which, when executed by a computer, cause the computer to: receive light field video data including a plurality of light field video sequences; display a first light field video frame of a first light field video sequence with a first viewpoint; receive an input indicating a viewpoint of the first light field video frame different from the first viewpoint; and in response to the indicating input, display the first light field video frame with the indicated viewpoint.

The computer readable storage medium can be a non-transitory volatile or non-volatile storage medium, such as, for example, a hard disk, an optical or magnetic disk or tape, a solid state memory device, etc. The storage medium thus tangibly embodies a program of instructions executable by a computer or a processing device to perform program steps as described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The term “light field video data” (hereinafter LF video data) used in the description should be understood as generally defined in the field, i.e. 4D video data comprising, at any instant, angular and spatial information about all the light rays at every point travelling in every direction in a space scene, and can be captured by, for example, a plenoptic video camera, a video camera array, or any other feasible camera device.

FIG. 1schematically illustrates an exemplary embodiment of an apparatus10configured to provide light field video data to a display device15. The apparatus10comprises a receiving unit11, an operation unit12, an input unit13and optionally a storage unit14.

The receiving unit11is configured to receive LF video data including a plurality of LF video sequences. Each frame of the LF video sequences has multiple selectable viewpoints. The LF video data can be captured and provided by a plenoptic camera to the apparatus10. In alternative, the receiving unit11can retrieve and receive the LF video data from a database (not shown).

The received LF video data can be optionally stored in a storage unit14, which can be any feasible and suitable memory device and can be either integrated in or supplementary to the apparatus10.

The operation unit12is configured to retrieve light field video frames from the receiving unit11or the storage unit14and to provide the same to a display device15. The display device15can be a display unit integrated in the apparatus10or an external display device outside the apparatus10. The display device15can be, for example, a television, a computer screen, a mobile device with display capabilities, etc.

The input unit13is configured to receive an input from a user or an operator choosing and indicating viewpoints of the LF video frames. A user can indicate a specific viewpoint of a certain LF video frame via the input unit13. In response to and according to the input instruction, the operation unit12is configured to retrieve the chosen LF video frame with the indicated viewpoint and to provide the same to the display device15. The input unit13provides an interface enabling a user to interact with and control the apparatus10. The types and the design of the input unit13can be flexible and varied for different demands and purposes.

Preferably, the operation unit12is further configured to provide multiple LF video frames temporally subsequent to the chosen LF video frame to the display device15in response to the indicating input, all of which being with the indicated viewpoint.

Optionally, the input unit13can be combined with the display device15. For example, the display device15can be a touch screen that displays the LF video frames and at the same time enables a user to enter an input. In this case, a user can indicate the viewpoint of a video frame directly using the touch screen15, rather than using a supplementary input unit13.

In one embodiment, the operation unit12is configured to provide a first LF video frame of a LF field video sequence with a first viewpoint and a second LF video frame of a second LF video sequence with a second viewpoint to the display device15. Preferably, the first and second LF video frames are temporally synchronized by the operation unit12.

More details about the operation of the apparatus10will be described below together with the proposed methods for displaying LF video data. Of course, the apparatus10can optionally comprise any other additional or alternative devices for the implementation of the embodiments of the following proposed methods.

FIG. 2schematically illustrates a preferred embodiment of the method for displaying LF video data. The LF video data discussed herein preferably includes a plurality of LF video sequences, and each LF video frame of the LF video sequences has multiple selectable viewpoints. The method comprises: receiving20the LF video data; displaying21a first LF video frame of a first LF video sequence with a first viewpoint; receiving22an input indicating a viewpoint of the first LF video frame different from the first viewpoint; and, in response to the indicating input, displaying21the LF video frame with the indicated viewpoint.

Referring toFIG. 1, one preferred embodiment of the method will be further explained in detail below using the exemplary embodiment of the apparatus10. It should be understood that the method can be implemented in any other suitable systems or apparatuses known and available in the field.

The receiving unit11of the apparatus10receives20the LF video data includes one or more LF video sequences from a camera or a LF video database. Each of the LF video sequences includes a plurality of LF video frames having multiple selectable viewpoints. For example, for an event such as a football game or a wedding, several LF video sequences relating to a same event will be available from different sources or cameras. Preferably, these LF video sequences are related to each other in certain content, and there are some semantical similarities among them.

The received LF video data can be optionally stored in the storing unit14and be used afterwards. The operation unit12retrieves the LF video data and the LF video sequences and frames thereof and provides the same to the display device15, e.g., a screen, to be displayed21.

Preferably, the default display of the LF video data shows multiple LF video sequences respectively. In particular, each of the LF video sequence is displayed21in a certain window or area on a screen15.FIG. 3(a)schematically illustrates one exemplary default display of multiple LF video sequences on a screen15. Sixteen LF video data A, B, C, etc., are shown on the screen15, where each one is displayed21in a respective rectangular window. The LF video sequences can be shown dynamically with a continuous video stream or be represented by one LF video frame of the corresponding LF video sequence. Preferably, the LF video sequences are displayed21all-in-focus in the default display. Of course, the amount of the LF video sequences shown in the default display is flexible and can be changed and arbitrarily chosen by a user for different demands and preferences.

In order to display one specific LF video sequence shown in the default display, a user can enter an input indicating a chosen LF video sequence via the input unit13of the apparatus10or directly via the screen15if it is a touch screen. The selection of one specific LF video sequence can be easily performed by, for example, using a mouse and a corresponding cursor on the screen15to click on the window of the target LF video sequence. When the screen15is a touch screen, the user can touch the window directly with his/her fingers. Any other existing and general remote control and/or user input devices can be used as well.

Referring toFIG. 3(b), the display of the LF video data “A” on the screen15is schematically illustrated, in response to an input indicating a selection of the LF video sequence A. A first LF video frame of the LF video sequence A with a first viewpoint is displayed21as a default setting, which is preferably the viewpoint closest to the average viewpoint of the corresponding LF video sequence. For example, the LF video sequence A can be displayed with the viewpoint A1as a default. Without a further input and/or instruction, it is preferred that the LF video sequence A is displayed frame by frame with the default viewpoint A1. In other words, the LF video data A is displayed as a video stream on the screen15with the viewpoint A1.

The multiple selectable viewpoints for a LF video sequence and the frames thereof are one of the advantages and improvements of LF video data. Accordingly, in the preferred embodiment of the proposed method for displaying LF video data, the viewpoint of a displayed LF video sequence is especially selectable and changeable. For example, as shown inFIG. 3, several selectable viewpoints associated with the video sequence A are shown on the screen15. In this example, sixteen viewpoints A1, A2, A3. . . are available for a user to select from. It should be understood that the amount of the available viewpoints is not limited and can be flexibly decided.

A user can enter an input indicating and selecting one specific viewpoint of the displayed first LF video frame of the LF video sequence A. For example, the input can indicate the viewpoint A7, which is different from the default viewpoint A1. In response to the reception22of the indicating input, the operation unit12provides the first LF video frame with the indicated viewpoint A7to the display device15. Then the first LF video frame of the LF video sequence A is displayed21with the indicated viewpoint A7on the screen15.

FIG. 4shows a viewpoint trajectory of the display of the exemplary LF video sequence A. At timing T1the first LF video frame of the LF video sequence A is displayed with the viewpoint A7according to the reception22of the indicating input as described above. At a following timing T2when a subsequent video frame is displayed, a user enters an input to indicate and select another viewpoint A13, resulting in the display of this LF video frame with the indicated viewpoint A13. If no further input is received, as shown inFIG. 4, the LF video sequence A is continuously displayed frame by frame with the last viewpoint A13. In other words, the LF video frames of the LF video sequence A are displayed in a temporal order with the indicated viewpoint.

According to the preferred embodiment of the proposed method, a user can change the viewpoint of the display of the LF video sequences at any time. It should be noted that the division of the screen15to show the selectable viewpoints can be either visible or invisible for the user. The dotted lines and areas marked in grey inFIGS. 3 and 4are used here merely for a clear explanation in the description and do not need to be visible on the screen15. A user can also select the viewpoints without the visualization of the viewpoints simply by clicking on and indicating a specific area or region of the displayed LF video frame corresponding to a specific viewpoint. For example, a user can be educated previously about the setting and visualization of the LF video sequences on the screen15, in which case the user would know how to enter an input to indicate a preferred viewpoint.

Alternatively, when the display device15is a touch screen, the user can select a new viewpoint using a swiping finger gesture on the input unit13. The processing unit12then determines the new viewpoint based on the direction and amplitude of the swipe gesture. Specifically, the new viewpoint is chosen in such a way that the direction of the viewpoint change between the initial viewpoint and the newly selected viewpoint matches the direction of the swipe gesture, and that the amplitude of the viewpoint change is proportional to the amplitude of the swipe gesture.

In addition to the selection and change of the viewpoints of the displayed LF video sequences, the focus of the same can also be chosen and changed. In one embodiment, the LF video sequences are displayed all-in-focus on the screen15, preferably as a default display of the LF video sequences. In another embodiment, a user can decide which object to be focused in one LF video sequences and/or where is the focus plane in the scene in the LF video sequences by indication operations similar to those described above. Specifically, the user can select one area or one object in the displayed LF video sequences by the input unit13or directly touching a touch screen15when it is feasible. This refocusing operation can be triggered by a selection of the “Re-focus” item shown on the screen15. A gyroscope can also be used in order to change the focus of the displayed LF video data.

In one embodiment, a user can also pause the display of the LF video data. For example, referring toFIG. 3, a user can trigger the pause operation by clicking on the “pause” item shown on the screen15. With the possibility to pause the display of the LF video sequences, it becomes easier for a user to change the viewpoints or the focus during the pause, which is especially interesting regarding a sport LF video or a LF video with changing content. The user can stop the display of a LF video sequence when an action of interest is performed, e.g., a goal attempt in a football game, and change the viewpoint and/or the focus of the displayed frame. In this way, the user can further enjoy the video LF data with a preferred viewpoint and focused region, which improves the viewing experience. Of course, the pause operation can be performed differently depending on different apparatus10, the input unit13thereof, and the display device15. For example, when the display device15is a touch screen, the user could pause the display by simply double touching the screen15.

In one embodiment, a function of stabilization can also be activated and/or triggered by a user, for example, by selecting the “Stabilize” item displayed on the screen15as shown inFIG. 3. In this way, angular information can be used to remove a shaky movement of a LF video sequence captured by a hand-held camera, which can be performed, for example, by using available methods in the field [II].

In one embodiment, the user can focus his/her attention on a particular object of interest in the scene captured by the LF camera. In this case, it is preferred to provide an operation for drawing a bounding box around a target object of interest in some reference LF views, preferably near the central viewpoint, and tracking the bounding box throughout several views, based on LF calibration data and visual tracking methods available and known in the field [III]. Relying on the object tracking information, the operation unit12of the apparatus10can generate from one target LF video sequence a monocular video where the object of interest is located as close as possible to the center of the displayed LF video sequence or to its initial position in the reference view. This can be achieved by dynamically selecting the appropriate viewpoint, among those made available by the light LF video data.

Specifically, the bounding box drawn by the user in at least one reference view provides a template image patch. This same image patch is also present in the other views of the LF data, but at displaced locations with respect to the reference view, because of the change of the viewpoint. The displacement of the image patch between the reference view and any other view available in the light field sequence can be estimated by methods known from the state of art and described in [III], such as normalized cross-correlation. Considering a given second view different from the reference view, adding as a vector the computed displacement for the second view to the location of the patch center in the reference view yields an estimate of the location of the template patch in the second view. In the proposed embodiment, the new viewpoint is computed by selecting, among all available views in the LF sequence, the view for which the computed location of the center of the template image patch is closest to the center of the image.

Furthermore, this bounding box, once tracked, may be used to automate the modification of the viewpoint in successive LF video frames. In other words, the new viewpoints in successive frames will be the ones on which the best focus can be found within the tracked bounding box. This way, the viewpoints will always be focused on the tracked object of interest.

In another preferred embodiment of the proposed method, the first LF video frame of the first LF video sequence is displayed in a main window on the screen15. The method further comprises displaying21a second light field video frame of a second light field video sequence with a second viewpoint in a side window on the screen.

FIG. 5schematically illustrates the described main and side windows displayed on an exemplary screen15in a preferred embodiment of the proposed method. In this exemplary illustration, one main window and seven side windows are shown, which are in the shape of rectangles. Of course, the size, shape, and the amount of the main and side windows are flexible and not limited. Accordingly, the amount of the LF video data shown in the side windows is also not limited and can be flexibly decided. Preferably, the size of the main window is larger than the size of the side windows, and the main window is positioned closer to the center of the screen than the side windows. In this way, the main window can be more easily seen by a user and attracts more attention than the side windows.

Preferably, the displayed first and second LF video frames of the first and second LF video sequences have similar content or at least some similar objects in the video sequences. In addition, the relative position of the main window and the side windows on the screen15is preferably related to the content of the first and second LF video frames displayed therein, respectively. For example, a second LF video frame X is displayed and placed at the right-hand side of the main window when there is a right displacement between the viewpoint of the first LF video frame displayed in the main window and the viewpoint of the second LF video frame X displayed in the side window. The side windows placed at the top, bottom or left-hand side of the main window are similarly arranged.

In alternative, the main and side windows can also be arranged according to other similarities between the displayed first and second LF video frames, such as colorimetry of the frames, global shapes similarities, time similarities, detected moves similarities, environment similarities, presence of same people in the frames as evidenced by face detection and face recognition, specified or detected objects similarities, etc. In addition, the similarities can also be defined and decided by the operation unit12of the apparatus10according to any suitable and feasible parameters and/or metadata acquired together with the LF video data.

In one embodiment, the first and second displayed LF video frames are preferably temporally synchronized, i.e. the displayed first and second LF video frames have a same timecode and/or have been captured at a same time. The synchronization of the displayed LF video frames can also be performed according to other characteristics of the video frames, such as capturing locations, people in the frames, audio, etc [IV]. The synchronization can be easily activated or deactivated by a user selecting the “synchronize” item on the screen15using the input unit13of the apparatus10or the display device15directly, similar to the activation/deactivation of other functions as described above.

Moreover, the first and second LF video frames can be optionally displayed all-in-focus, preferably as a default display. The focus of the displayed LF video frames can be arbitrarily changed and selected as described above with the use of the “Re-focus” item shown on the screen15.

Referring toFIG. 6, in one preferred embodiment, the method further comprises: receiving22an input indicating a viewpoint of the displayed first LF video frame in the main window; receiving23an input selecting the displayed second LF video frame with the second viewpoint in the side window; and in response to the selecting input, (a) displaying21the second LF video frame with the second viewpoint in the main window, replacing the first LF video frame, when the second viewpoint is in close proximity to the first viewpoint, or (b) displaying21the first LF video frame in the main window with a changing viewpoint when the second viewpoint is not in close proximity to the first viewpoint. The transition from one LF video data to another LF video data is denoted as “Video Jump” hereinafter, of which the transition path is schematically illustrated inFIG. 7in detail.

FIG. 7(a)illustrates the first situation, where the second viewpoint of the second LF video frame (right-hand side) displayed in the side window is in close proximity to the first viewpoint of the first LF video frame (left-hand side) displayed in the main window. At the time instant T1, the first LF video sequence is displayed in the main window on the screen with a specific viewpoint (marked in grey in the figure). At timing T2, the displayed viewpoint of the first LF video sequence is changed by using, for example, the method described above. The changed viewpoint is then in close proximity to a viewpoint of the second LF video frame of the second LF video sequence, which is displayed in a side window next to the main window on the screen15. In this case, in response to a user input selecting the second LF video frame, the corresponding second LF video sequence can be displayed in the main window, replacing the first LF video sequence. Accordingly, at timing T3, the second LF video sequence is displayed in the main window (possibly with a same second viewpoint or a further changed viewpoint as shown in the figure), and the first video sequence is preferably displayed in a side window instead.

The selection of the second LF video sequence can be performed, for example, via the exemplary input unit13of the apparatus10. In the case where the display device15is a touch screen, a user can simply click on the side window displaying the second LF video sequence and sweeps it to the main window. It is preferred that the Video Jump operation is seamless, which means that the display of the first LF video sequence in the main window is not paused during the transition of the display from the first LF video sequence to the second LF video sequence. A gyroscope can also be used for such a seamless Video Jump operation.

FIG. 7(b)shows another situation, where the second viewpoint of the selected second LF video frame of the second LF video sequence is not in close proximity to the first viewpoint of the displayed first LF video frame of the first LF video sequence at the time of the selection of the Video Jump operation. In this case, it is still preferred that the transition of the LF video sequences is performed smoothly in terms of viewpoints. That is, the first LF video sequence is displayed in the main window with a changing viewpoint until the viewpoint of the first LF video sequence is in close proximity to the second viewpoint of the second LF video sequence. As shown inFIG. 7(b), the Video Jump operation is triggered at T, resulting in a changing viewpoint of the first LF video data throughout the timings T+1, T+2 and T+3. At the timing of T+3, the viewpoint of the first LF video sequence becomes in close proximity to the second viewpoint of the displayed second LF video sequence, which leads to the display of the second LF video sequence in the main window as described in the previous situation.

In one embodiment, the Video Jump operation can also be performed with the pause function as described above. For example, in the case of a goal attempt in a soccer game, a user might want to watch other LF video data from other viewpoints capturing the same goal post. In this case, the Video Jump is also possible to be operated within the pause mode. In other words, the Video Jump operation can also be performed with the pause operation. The transition of the LF video sequences is still preferably performed smoothly in terms of viewpoints. When the LF video is played again, the visualization starts from the last LF video sequence watched.

In one embodiment, the proposed method for displaying LF video data can be used integrated with available broadcasting techniques and methods. For example, when a user can get access to a set of broadcast LF videos that are related to a same live event, e.g. a soccer match, a Formula 1 Race, a stage of the Tour de France, etc., the proposed method can be applied and implemented provided that the display apparatus embeds multiple tuners and time-shifting features. The display and navigation among the several broadcast LF video signals is possible, as well as the “pause” and other operations mentioned above. One exemplary case is that when the user browses the LF videos or manipulates them during a “pause” operation, the broadcast live LF video signals can be recorded and played back or resumed at a later time.

In one embodiment, the proposed method for displaying LF video data can be used by a content provider to edit broadcast or broadband content. On the back-end side of a professional services provider, many LF video data can be used as inputs to edit final LF video data. These final video data can then either be stored for later broadcast/broadband or can be broadcast on the fly, with maybe time shifting. Furthermore, once final LF video data have been edited the end-user, professional as well as consumer one, can either receive the final LF video data on a broadcast channel or request for its playing on demand via a broadband channel.

REFERENCE