MULTI-CAMERA MULTIVIEW IMAGING WITH FAST AND ACCURATE SYNCHRONIZATION

There is provided a method comprising: receiving a communication from one or more devices capable of recording content, determining, using a wireless communication transceiver, a geographical location of the one or more devices, determining an orientation of the one or more devices, receiving content capturing an event and recorded on the one or more devices, storing the content capturing an event and recorded on the one or more devices, and creating, from a collection of recordings comprising at least the stored content capturing an event and recorded on the one or more devices, a single representation of the event by combining segments of the collection of recordings.

FIELD

The present disclosure is directed to methods and systems that can generate an amalgamated recording of an event. In particular, the present disclosure includes methods and systems for synchronizing videos from multiple devices based on localization data obtained from a communication transceiver.

BACKGROUND

The field of view in a single multi-view lightfield camera is limited by the size and properties of the main lens. Thus, in many events a collection of multiple cameras are placed at known locations and coupled with advanced computers to process the large amount of data produced by the cameras.

In existing systems, the multiple cameras are synchronized. A variety of methods have been created to accomplish this task. For example, audio and video content have been jointly used for multi-camera synchronization. Image feature based synchronization and epipolar lines have also been considered. Another method is known as SocialSync, in which first the capture timestamp is estimated based on timestamps of previously delivered frames. Furthermore, the timestamp is improved by attempting several preview sequences until alignment is obtained. Managing a network of cameras is another issue that needs to be considered. The localization of a camera in a network has garnered considerable attention. For example, dense scene matching to improve the 6 degrees of freedom camera pose estimates is known in the art. Even though streaming platforms supporting real-time synchronization of multiple cameras exist, they are encumbered by their reliance on computer vision based localization. Geo-tags have been used to improve 3D models created from crowd sourced image datasets. Strategies for scheduling collection of sensor data have also been used. The present invention determines camera orientation and position without computer vision based techniques, creates real-time interactive 3D visualization without 3D model creation, and builds a collection of individuals acquiring images and videos of events in a gamification environment.

Although there is considerable work in using a collection of cameras to create multiview images and videos, existing work seems to focus on reconstruction shapes from several viewpoints, tracking humans from one camera to another, or managing a collection of cameras. In contrast, we propose a new system to determine the location of cameras relative to each other and to do so very quickly, without relying on computer vision, or image matching techniques. Several of the technologies that are needed to enable the new method for camera localization and synchronization have been shown to be feasible. For example, sub-nanosecond time of flight is possible on commercial WiFi cards; precise positioning using LiFi has been demonstrated for industrial applications; and direction of emission of a WiFi signal has been utilized for several applications. It has been noted that that signal processing can be used to discriminate LoS/NLoS components of wireless signals, and by applying Angle-of-arrival/Departure (AoA/AoD) methods to only the LoS component, a precise orientation can be obtained.

SUMMARY

According to an aspect, there is provided a method comprising: receiving a communication from one or more devices capable of recording content, determining, using a wireless communication transceiver, a geographical location of the one or more devices, determining an orientation of the one or more devices, receiving content capturing an event and recorded on the one or more devices, storing the content capturing an event and recorded on the one or more devices, and creating, from a collection of recordings comprising at least the stored content capturing an event and recorded on the one or more devices, a single representation of the event by combining segments of the collection of recordings.

In this manner, a multiview recording of an event may be created by incorporating user-generated content. The transceiver-based localization of the user devices ensures fast and streamlined creation of the multiview recording. Users thus receive a recording of the event enhanced with different perspectives.

For example, a plurality of users may use their smartphones, or other user devices, to capture video at locations such as a stadium, arena, or tourist sightseeing spot. The above method allows a video captured by a first user to be augmented by appending at least a portion of a video captured by a second user, for example, to provide video of an event or angle that was missed by the first user.

Additionally, in some embodiments, the method further comprises analyzing the collection of recordings for negligible content, such as irrelevant or redundant material, and removing the negligible content from the collection.

In some embodiments, the single representation includes a synthetic image created by combining aspects of the collection of recordings.

In some embodiments, the method further comprises analyzing a static background of the content capturing the event to determine how desirable the content is to a created single recording.

In some embodiments the one or more devices comprises a smartphone. In some embodiments the one or more devices comprises a static camera. In some embodiments, the one or more devices comprises a device mounted on a drone.

In some embodiments, the geographical location of the one or more devices is determined based on decentralized communication among the or more devices and a second device.

In some embodiments, creating a representation of the event further comprises integrating the segments of the collection of recordings with a live broadcast.

In some embodiments the method further comprises incentivizing users to record content of the event. In some embodiments the incentivizing incentivizes the users of the one or more devices to record a desired view of the event.

According to another aspect, there is provided an apparatus configured to perform any of the methods set out in the above aspect. An example of such an apparatus comprises control circuitry configured to: receive a communication from one or more devices capable of recording content, determine, using a wireless communication transceiver, a geographical location of the one or more devices, determine an orientation of the one or more devices, receive content capturing an event recorded on the one or more devices, store the content, and create, from a collection of recordings comprising at least the stored content capturing an event and recorded on the one or more devices, a single representation of the event by combining segments of the collection of recordings.

DETAILED DESCRIPTION

Methods and systems according to the present disclosure allow content of an event generated by a first device to be combined with content of the event generated by a second device to create a recording of the event that encompasses multiple views. In the exemplary embodiments set out below, video captured by a first device is augmented using video captured by a second user device or multiple additional devices. In other embodiments, such methods and systems may be used to augment content other than captured video, such as audio, visual, multimedia, or other content generated or captured by the first user device. Further additional embodiments may be used with content captured by a wide range of devices and systems.

FIG.1shows an overview of the system described herein. The system comprises devices101,102,103through N. Each device is capable of recording content via camera110, or other content capturing mechanism such as a microphone. Each device101also has a processor111, a wireless transceiver112, a memory113, and a user interface116. Devices101may be cameras, smart phones, static cameras, mobile cameras mounted on drones, AR glasses, or any other device that can capture content. In a preferred embodiment, one or more of the devices may support lightfield imaging. In this case, the configuration of multiple cameras or smartphones can collectively create a multiview lightfield-imaging environment. More specifically, while the multiple cameras allow large movements in view like looking at a person from front and behind, lightfield support in individual cameras allow finer movements of individual objects or create small view changes that are suitable for 3D TV displays or generating stereo displays. The devices are connected to wireless communication transceiver104via communication link105. Data sent and received by wireless communication transceiver104enables system150to locate devices101through N in space which ultimately allows content from the different devices to be pieced together quickly. The devices are also connected to the cloud106via a second communication link107. Communication link104and107may be the same. For example, both may be a Wi-Fi connection. Communication links may further be WiFi Direct, Peer-to-Peer, or other means, or combination of means, for connecting devices. System150further includes Cloud106, a network connecting the mobile devices101through N to the server108. Server108may be, for example, servers associated with an application that the mobile devices use to upload the content. The cloud106is further connected to server108via communication link109. The server108comprises a wireless transceiver112, a processor114, and a memory115. The server108is in communication with viewer117via an additional communication link118.

FIG.2depicts a flow chart of an exemplary method of they system150. In a preferred embodiment, a device connects with a transceiver104. The transceiver104sends a communication signal to the device101at step201through which it determines the position of the device101at processor114at step202. They system150next determines the orientation of device101through inertial sensors or other means at step203. The device101next collects content of an event via camera110and stores the content in memory113at step204. The device then uploads the content to the cloud106via communication link107along with the determined position and orientation information at205. At step206, this information is transferred or downloaded from the Cloud106to the server108which receives the information along with that of at least a second device102. The content collected from the various devices makes up a collection of recordings from which the server creates a final recording, the amalgamated recording, which is an amalgamation of the collected content. At207, the processor114of the server108processes the information, that is the collection of recordings, orientations, and locations of the devices, to create an amalgamated recording. The server108and cloud106may be part of a third party service. For example, in some embodiments, a third party may be responsible for collecting the content and creating the amalgamated recording. A television provider may be a customer of the third party in another embodiment. In one embodiment, the television provider allows its video player to communicate with the third party content server via an API to retrieve content and play the content in its video players. The amalgamated recording comprises at least segments of recordings from the device101and at least a second device102. Here, the segments may be clips of the recordings, aspects of the recordings, that is information derived from the recordings such as position or alternative views of an object, or images from the recordings. The processor114then distributes the created amalgamated recording to viewers117for consumption at step208. In other embodiments, the amalgamated recording may be distributed to viewers117by another means, such as, for example, fetched via API calls by a service (such as a playback service) associated with the viewer device117and/or an application installed on viewer device117. Finally, at step209the system150analyzes the amalgamated recording to determine contribution of the device101.

As outlined above, the described invention combines content, preferably video, of an event where the content is captured by multiple devices, device101through device N. The content from the multiple devices, device101through device N, is combined to create an amalgamated recording of the event that combines aspects of the individual recordings from the multiple devices to generate a recording with multiple views of the event. The described system relies on localization of the contributing devices to synchronize the multiple views. This localization is based on the devices' communications with a communication transceiver104such as a cellular wireless base station, a WiFi or LiFi access point, or similar. Transceivers may be static or mobile. Mathematical optimization techniques can be used to obtain the best possible estimates of device locations from an over-determined system of equations. In one embodiment of the proposed system, initial estimates of localizations of devices101are obtained from built-in GPS and inertial sensors, which are now widely deployed in mobile devices including smartphones. In another one of the preferred embodiments, some of the devices101are on Wifi while others are on cellular networks. In this configuration the devices101that do not have WiFi will acquire position information by first using GPS data and then refining estimates using computer vision based matching. Furthermore, orientation information can be acquired using the background matching approach discussed below. The devices101having only cellular communication may still access the data relating to the location and popularity scores of other cameras, as well as the user interface to help guide them to better locations for capturing images or videos in real time.

Location estimates of the devices101derived from transceiver communication105can be improved by creating virtual views for static backgrounds, for example, where the virtual views are preferred views synthetically generated using available views of the event and simulating small camera rotations. Then, matches between actual and virtual views can be used to refine localization estimates for devices101in a network. Location estimates may also be improved by comparing backgrounds of neighboring devices101to infer additional information.

FIG.3Ashows a configuration of devices, capturing content in communication with a communication transceiver104. The configuration allows positions and orientations (for example, distance and direction) of the devices to be determined relative to a central location. Here, a central transceiver301, a communication transceiver104, sends directional wireless data302to various devices310,311,312,314, and315, to determine the position and orientation of each of these devices relative to the transceiver301. Following this step, the relative positions and orientations of all the devices relative to one another can be determined. Such fixed points, with known “absolute position” can be used to calculate the positions of the devices. Furthermore, multiple such fixed points can be used to improve the localization of devices, known as triangulation, which is well documented in the art.

FIG.3Bshows a configuration that allows positions and orientations to be determined by a collection of decentralized devices,310,311,312,313, and314, where some devices may be static and other mobile. Here, one smart phone, for example310, may determine its position and orientation relative to nearby devices, for example311and314, and this information is circulated for a collection.

Referring now toFIG.3C, a more general form of interaction among a collection of devices such as cameras316, smart phones317, drones318, and centralized transceivers319is shown. In this configuration, all devices may communicate with one another. In all instances, the devices discussed above may be static or may be in constant motion. For this reason, the location of the device101may be continually determined, determined for each image recorded, or for a specific period of time. The multiview image or video data is uploaded to the cloud106via communication link105, along with the position and orientation information, where it will be processed by the processor114of the server108to create the amalgamated recording. The server108then distributes the amalgamated recording to viewers117via communication link118.

FIG.4depicts a scenario in which a first user400generates content by capturing video using a first user device101. In this particular example, the first user device101is a smartphone, and the first user400is making a video recording408of an event401, here a soccer match, from their location404within a sports stadium406. The recording408covers a field of view410during a particular time interval. For example, the recording408may capture action in the match and/or images of other attendees in the stadium406, such as friends and family members of the first user400.

Other attendees and devices101, such other smartphones, television broadcast equipment, or camera operated by the game officials, are likely to also capture recordings of the event401in sports stadium406. As each device101presents a different view or perspective of event401, and is limited by that view and the frame of the camera of the device101, it may be advantageous at times to combine recordings from the various devices, device101through device N, into an amalgamated recording that can display a wider or additional view of the event. For example, in capturing a soccer match, device101might have a view of a player kicking the ball, while a second device102might have a view of the trajectory of the ball. Both of these perspectives are interesting to viewers117and therefore valuable contributions to a representation, i.e., recording, of the event401. The amalgamated recording has the ability to include both perspectives because it includes content from devices capturing both of these perspectives. The system150further has the ability to create multiple amalgamated recordings. The system150may also in some embodiments process overlapping perspectives, typically from multiple devices, device101through device N, to create a new, synthetic viewpoint from a desired viewing plane. In the present system, a synthetic image or video may be created where there is sufficient coverage of an event. A 360 degree video, or surround video, of objects is not necessary to generate the synthetic image or video. The system150allows the user to choose from multiple amalgamated recordings, each of which is derived from one or more user device recordings. In some embodiments the viewing plane may be chosen by the system150. In some embodiments, a viewer117may choose from multiple viewing plane options determined by the system.

In another example, a camera operated by a sportscaster operating a first device101may have a close up view of the field while a fan has a partial bird's eye view of the field from the stands. The amalgamated recording of both devices, plus potentially additional devices, has the ability to include both of these perspectives of the event as well. In some embodiments the amalgamated video includes the option for viewers117to view different perspectives of the event401, and interactively move between those perspectives. In a preferred embodiment, the system150can associate the amalgamated recording with a live broadcast recorded on a television camera. The amalgamated recording may be integrated with or accessible to the live broadcast, where integration comprises combining the amalgamated recording with the broadcast to create a representation of the event that includes aspects of both the live broadcast and the amalgamated recording, or offering the amalgamated recording as supplement content to the live broadcast. The amalgamated recording in some embodiments may be synced with the broadcast in real time or near real time. In another embodiment it may also be offered as a subscription (e.g., users subscribe to receive videos associated with specific teams, camera angles, player, etc.) with the broadcast and served as a feed (e.g., separate feed from the broadcast) of additional content.

The system150creates the amalgamated recording by receiving content, orientation, and position data from the devices101. Position data allows the system to process varying perspectives without cumbersome artificial intelligence processing, although in some embodiments, computer vision based feature matching is used to improve the accuracy of the estimates of orientations of one or more devices101. In another embodiment, estimates of distances to 3D objects can be obtained by a number of means, including depth from focus, the use of built-in ToF, Lidar in SmartPhones, or RGB-D options in cameras/smartphones. These depth estimates allow approximating the pixel properties regions of different camera images that can be used to interpolate a region of a viewing plane.

Referring now toFIG.5, a system and method to estimate an image or video from those captured by nearby cameras is shown. Here, devices501,502, and503have a view of a goal in a soccer match. However, none of the devices provide a head on view. device501has a view above and behind the goal, device502sees behind the goal, and device503has a view of the goalie in front of the goal but not of the goal itself. An image or a video frame for a head on view V needs to be estimated. Such a view can be created from content captured by devices501,502and503. Note that system150does not need to create 360 degree video, or surround video of objects. All that is needed is sufficient coverage of important activities in a scene.

Referring now toFIG.6, the boxes601,602, and603in this figure show the orientations of the imaging planes for devices501,502, and503respectively, and the viewing plane V1for view V. While these planes are shown as rectangles in 2D in this diagram, in reality these would be 2D rectangles representing the viewing plane in a 3D environment.

Referring now toFIG.7, the imaging planes inFIG.6are transformed to701,702, and703respectively via processor114to make all image planes parallel to the view plane V1for view V. The transformed image planes, like the original image planes, are 2D rectangles.

Referring now toFIG.8, an illustrated method to compute the pixel values in a region R of the viewing plane V1is shown. In this example, the region R represents a portion of view V on the viewing plane V1. A portion of a view may be a subset of the view, represented by a two dimensional shape, smaller than or equal to the two dimensional shape which makes up viewing plane V1. This portion can be interpolated from the regions R2and R3, that were captured by device502and503respectively, and that capture objects and views within view V. Extrapolating data about objects and views in R2and R3allows the system to generate a view R. Repeating this process for other portions of view V, and combining those segments, allows the system to generate a synthetic view V.

With the collected content and location information, and any supplemental data, the system150creates an amalgamated recording featuring multiple perspectives captured by multiple devices101. Devices101capturing content can submit their videos to an online platform to create spatial video. The system150is able to interpolate views and create spatial video. As discussed above, a requested perspective (such as in a VR HMD) may be generated provided sufficient number of views have been captured that may be interpolated to the requested view.

In a preferred embodiment, the locations, orientations, and relative contributions of all the devices101that are currently active are updated in real time and made available to all other devices101connected to the system150. This enables real time gamification of multi-camera content capture. In this mode of operation, users with devices101can move positions in the environment and, by viewing relative contributions, maximize the popularity of their views. It also allows additional information from a centralized server108to reduce overlap between views and cover important regions that are currently not covered by any of the devices101. In some embodiments, redundant position and orientation data may be generated as a result of the device interactions for a collection of devices collectively capturing a scene with multiview. The system150however may use this redundant information in obtaining better localization accuracy for all the devices101. In a preferred embodiment, directional signals105are adapted based on the current positions of moving devices101. This allows higher efficiency and accuracy in determining the positions and orientation, and thereby higher quality experience for viewers117. Given the knowledge of current positions of devices101, the directional wireless signals105can be concentrated primarily around the regions of these devices101and not uniformly in 3D space to allow much finer angular direction estimation and 3D localization.

In some embodiments, the system150processes the collected recordings and, using the background images, removes redundant and irrelevant recordings as negligible material.FIG.9provides a method for determining and eliminating redundant devices. Each device101in consideration is numbered on a list of devices. At step901a device is chosen and its number is designated first on the list. At step902system150checks the viewing area of the device relative to a panoramic background scene, which is a preferred background representing the region the system150will cover. If the viewing area matches the panoramic background scene, the method moves to step903. If it does not match, the device is dropped from consideration. At step903the system150determines if the viewing area of the device is covered by one or more closer devices. If it is, the method moves to step904, where the device is dropped from the list of devices considered. If it is not, the method moves to step905. At step905the device remains on the list for consideration and the first device number is renumbered by one. Finally the method moves to step906which determines if any additional devices remain for consideration by determining if the first device number is higher than the number of devices. If it is, the method ends. If it is not the method returns to step902. In some embodiments, when a device101produces negligible or suboptimal content, all content from the device101is removed from consideration into the amalgamated recording. This approach reduces the overall complexity of the multi-view capture environment while maintaining the same quality. Furthermore, devices101that contribute irrelevant content can be suspended from contributing content for a time period. As a pre-processing step at processor114relevant frames may be added to a queue, while frames that are irrelevant to the event could be discarded. Furthermore, uploads from devices101that are past a given distance from a point of interest can be limited as well. For example, viewers117might not be interested in frames from the south end of the field in a soccer match if the ball is in the north end. This strategy will enable faster processing of relevant spatial video at processor114.

Referring now toFIG.10, an architecture that can be used to improve the localization of multiple devices101in a multiview capture network is shown. On the left side is presented a collection of images1001from devices with known locations. On the right is an image1002from a device101with an unknown location. The background of image1002is similar to those of1001. The location of a device101with a similar background to that of the images801from known devices can be improved by comparing the backgrounds of all images. The similar backgrounds indicate that the device101that created the image802is in close proximity to the device or devices which captured the images1001. By analyzing shared objects,1010and1020, using a neural network and distribution learning, the system150can infer further information about the device101that created the image1002. Even though the localization and synchronization of cameras can be achieved purely by directional wireless signals and time of flight (ToF) sensing, these estimates can be further improved by detecting the static background in a video and matching the background in the scene after compensating for rotational movements of a camera, such as pan, tilt, zoom, and translation. Given the background images from two nearby cameras, the best match will be obtained when the correct relative positions and orientations between the devices101are known. Assuming that the estimates based purely wireless signals and ToF may have slight inaccuracies, images with small rotations can be synthetically generated from an existing image. These synthetic images can then be matched with actual images to determine the best match and thereby the best estimates of positions and orientations of cameras.

In some embodiments, the system may also use inter-device feature communication for more accurate localization, faster and precise video synchronization among cameras. In another embodiment, the system may use Active Calibration and extend distribution learning to a network of cameras to determine foreground and background using distribution learning to robustly calibrate for camera parameters in real-time. The rotational parameters that are needed for active calibration are obtained directly from the WiFi/LiFi based fast camera localization. This avoids the need for complex calibration algorithms and the need for having some known shapes or patterns for calibration.

Referring now toFIG.11, an architecture for extending distribution learning using moment distribution is discussed. Here, in addition to arithmetic distribution layers of addition and multiplication, moments of various orders are considered. These moments can be added as new layers in a neural network. Given any arbitrary function, it can be expanded using a Taylor series and approximated by dropping higher order terms. Thus, even arbitrary functions can be approximated by moments of lower orders.

In further embodiments, changes in camera parameters like focal length modifications over time resulting from zooming in or out, can be calculated in real time by keeping track of changes in camera orientation and using active calibration techniques considering features of the background scene. In one embodiment, the device101(e.g., smartphone camera) provides data to the server108on the image zoom along with position and orientation. If it is not available from the device101, zoom may be determined by first building up the overall background of a scene based on background subtraction and image mosaicking. Following this, as cameras or other devices101move around, their precise orientations can be improved (over value derived from LoS/AoA method) by matching the current image or video frame captured with a part of the static background. Furthermore, the zoom factor of a device101can be determined by the size of the area of the background image captured. In another embodiment, the wireless transceiver104responsible for position and orientation determination of participating devices101through N, may be allowed to set parameters such as zoom, resolution, etc., in the user devices101through N. This reduces parameter variation across devices easing the processing required to generate amalgamated recordings.

In some embodiments, a user400collects the recording via a user interface116installed on the user device101. The user interface116may provide a display for the images captured by the device101and provide options to record, upload, delete, edit, or add effects. It may also include a feed or images of the event401comprising at least a portion of the amalgamated recording or a separate feed or images. In the scenario of a live broadcast, the amalgamated recording can be made available to viewers117watching an event401via a broadcast, a breaking news event getting live coverage, or a replay of the broadcast on individual devices.

In some embodiments the user interface116encourages user participation. This system measures how much one camera or device101view is observed relative to others. The system also accounts for view interpolation, where more than one device101view may be used to interpolate an intermediate view. In some embodiments, the system encourages user participation in the form of revenue share of the amalgamated recording a device101has contributed to. In some embodiments, the contribution, and by extension the popularity, utility, or revenue share of a user, is determined by calculating the number of time intervals (minutes, seconds, etc., for example) and dividing that number by the total time intervals of the amalgamated recording to arrive at a relative contribution. When creating a synthetic view based on a viewer location and interpolated views, the relative proportion of the view from a given device that is used can be determined and added as the contribution from this given device for the given time interval. In a preferred environment, the total contribution from all the devices for a given time interval for an interpolated view will be adjusted to represent a percentage, or add up to 1 or other fixed pre-determined constant. This adjusted representation may be used in calculating contributions to the final recording.

The contributions can be accumulated over shorter or longer durations to estimate contributions over a specific period. In a preferred embodiment, the current popularity values for different devices101are calculated by the processor114of the server108based on tables storing usage of content by viewers117in real time. These values are displayed and updated in the user interface116in real time to all people capturing content on a device101. In certain circumstances, or during an initial period of time, sufficient or meaningful viewer data may not be available. For these situations, historic tables built using viewing data from past events can be used. The historic tables can provide data on content capture locations and movements of capture devices with respect to a fixed location, preferred orientations and changes to zoom. In one embodiment, the system assigns scores as devices upload content. In a preferred embodiment, uploads from devices101with a high relevancy scores are prioritized. Devices101can be also prioritized based on the device capabilities or specifications. Device priority can also be based on distance to the field/goal or the location of an action, for example, the closer the device101is to event401the more likely its content will be selected.

In some embodiments, the user interface116encourages users400to access a preferred perspective. In one embodiment, the user interface116displays a representation of the popularity or utility of the current perspective a user is capturing. In some embodiments, a relative popularity value is displayed to signify this representation. For example, inFIG.12the values shown next to C1 to C8 are scaled by the total to get a relative value that adds up to 1 for all the values. Cnew represents a new participant who is trying to find a good position. In some embodiments, the relative popularity value will change in the user interface116as the user changes positions and perspectives. In this figure, Cnew is looking for the highest valued view by observing the popularity values at the cameras in different perspectives. The values and locations/orientations for different device101can be displayed in user interface116using a 2D or 3D graphical user interface to assist active participants to move around more easily. The user interface116may further include directions to instruct users400to find the most optimal view. For example, the user interface116may send a message recommending that the user400move to the left or right, move the camera up or down, or zoom in or out. Even though we discuss Cnew moving, other participants like C1 to C8 may also be constantly in motion in real time using this system and method.

In preferred embodiment, a user400can specify the trajectory of his or her device101by touch and/or gesture interactions on the user interface116. In another preferred embodiment, a user400may be prevented from moving or turning to a location where no view is available. In another preferred embodiment, users400see an error notice or missing region when they move or turn to a location where no camera view is available. In another one of the preferred embodiments, user400sees a virtual environment based on knowledge of surrounding regions when they move or turn to a location where no camera view is available.

The interface116can be modified to provide guidance to move to locations that are missing from the recorded content, need additional content, or that are more popular with viewers117. Locations that are missing from the recorded content, need additional content, or that are more popular with viewers may first be determined by the system150or otherwise communicated to the system150. The system may in some embodiments then communicate this information to the user interface116. The user interface116can also be modified to provide guidance on modifying the orientation and zoom of a device101without moving location; a strategy that can be useful when it is difficult to move location, e.g., given a fixed seat in a stadium. In another embodiment, a user interface116may provide information on current relative popularity of other devices101and their locations/orientations.

The user interface116can provide still additional features on the viewing side as well (i.e., on viewer device117). For example in a preferred embodiment, the user interface116may flag videos featuring a viewer's117favorite player or mascot. In another one of the preferred embodiments, the user interface116includes a progress bar and the a progress bar is updated throughout a game (e.g., by overlaying various icons on the progress bar) to include or highlight references to most viewed and most liked spatial videos. In another one of the preferred embodiments, viewers117may subscribe to content captured from one or more specific locations relative to the game field, for example, a viewer117may want to see a game from a home or visitor perspective. In another one of the preferred embodiments, supplemental content is available for objects in view and these objects are highlighted or marked to make viewers117aware that supplemental material is available for the marked items. Viewers117on their own display devices can request this supplemental content. This feature can also be useful when an action happens during an ongoing event. For example, if two players get into an altercation during a goal and receive a penalty for the altercation, a viewer117might want to watch the altercation to understand the reason for the penalty. In this situation, the system150may have additional content on the altercation, mark the players involved as having additional content, and the viewer117may choose to view the additional content on his or her display.

In another one of the preferred embodiments, the user interface116on viewer device117adds an “as seen by fans,” “as seen by audience,” or similar, icon to the amalgamated recording when viewers are watching content captured by user devices, which may be one or more of devices101through N. This supplemental content can be displayed alongside a main event or broadcast. In another embodiment it can be displayed in a PiP (Pictured in Picture, for example) or it can be displayed in a separate application simultaneously with the main event or broadcast. This feature is accomplished by sharing metadata of content available at sever108with viewer117and the metadata may be deeplinked to the content itself (i.e., to the video). The icon may be displayed on the user interface116based on the availability of content or metadata associated with the content at server108. For example, metadata such as time of capture or location of capture of the content captured by devices101through N can be correlated with the time of one event in a larger event, such as a goal in a soccer match. Viewers117may in some embodiments subscribe to certain content or type of content. The metadata associated with such content may include additional information such as indicating that a video contains images of a player (e.g., L. Messi, T. Brady, etc.), a play or event (e.g., penalty kick), or a team with possession of the ball. The metadata may further indicate a detailed location such as where content was captured within the stadium or field (south end of the field, north end of the field, etc.). Such information can be derived based on the location of transceiver104. The “as seen by audience” or similar feature can also be dedicated to automatic replays or replays initiated by a viewer117watching on his or her own display device. In some embodiments of the present invention, system150, unlike existing systems, constructs new and different formats, such as those from new perspectives and multiple formats and automatically associates them with an action (e.g., pass interference, touch down etc.) that occurred earlier within a particular event or game by correlating the time of the event with the time a video (i.e. content) was captured. The association may also be based on the location of device101and the location of the event. Such association can be accomplished in real time or near real time. In embodiments where a single representation is displayed in response to a replay request, the content can be referenced in a manifest file and used in trick-play requests. This means that in those embodiments, a replay will not repeat content seen in the broadcast, but instead displays content captured by the devices101that is processed at server108.

FIG.13shows an additional method of one embodiment of the invention. The method comprises the system150, at step1301, receiving a communication from one or more devices, device101through device N, the devices being capable of recording content. Device101represents a first device and device N represents an Nth device where N is the number of total devices in the system150. Step1302is determining, by system150and using a wireless communication transceiver104, a geographical location of the one or more devices101through device N. At step1303the system150determines an orientation of the one or more devices101through device N. At step1304the system150receives content capturing an event401and recorded on the one or more devices101through device N and at1305stores the content capturing an event401and recorded on the one or more devices101through device N in a memory113. Finally, at step1306the system creates, at processor114, from a collection of recordings comprising at least the stored content capturing an event401and recorded on the one or more devices101through device N, a representation of the event401by combining segments of the collection of recordings.

FIG.14shows yet another method of another embodiment of the invention. The method comprises the system150providing a user interface116displayed on a user device101at step1401. At step1402it receives one or more recordings of an event401from the user device101and at step1403determines, using a wireless communication transceiver104, the geographical location of the user device101. At step1404the system150determines the orientation of the user device. At step1405the system150stores the recordings of the event401from the user device. At step1406it creates a representation of the event401from a body of collected content wherein the body of collected content comprises at least the stored recordings of the event401from the user device101. At step1407the system150analyzes the representation to determine the contribution to the representation from the stored recordings of the event401from the user device101and step1408incentivizes, via the user interface116, contribution to the representation from the user device101

In one embodiment, server108processes content (e.g., combines portions of different, for example, videos recorded by user device101through user device N) based on context. For example, the context can be a football game, a basketball game, a baseball game, etc. The context can also include additional information, for example, teams, location of the game (e.g., location of stadium), time the game starts, time the game ends, time the teams are taking a break (e.g., half-time), etc. In some embodiments all recorded content uploaded from the devices is associated with a context. For example, the content can be tagged to a game or event, and later analyzed to add additional tags (e.g., identify an action in the game—such as a touchdown, interception, penalty kick, identify a team that has possession of the ball, identify a player or players involved, etc.). The additional tags allow serving content of interest or based on user preferences. For example, viewer device117might subscribe to receive content from specific devices through server108based on predefined tags or criteria (e.g., penalty kick, touchdown, etc.). Analysis can be done by utilizing trained machine learning models to generate automatically generated additional tags.

In one embodiment, a software application on device101allows the selection of a context. For example, the application may be associated with a service that processes and generates amalgamated recordings and/or makes the content available to viewer device117. The app may have different interfaces for devices101and viewers117or there may be one app with both recording and viewing features. More specifically, the application might present a user interface116that allows the user to confirm at least one context associated with the content that is to be captured and automatically uploads such video to server108. In some embodiments, the application may suggest the context, such as a game, “Seahawks vs. 49ers,” based on the location of the device101and enable the user400to authorize the automatic upload of the content. In one embodiment, the capture and recording process takes place from within the application (i.e., the application presents a UI that includes a record button).

In one embodiment, server108polices the content uploaded by devices101through N. For example, in one embodiment, devices101through N are only permitted to upload content if the device location matches the location of a live event (e.g., live football game). The location of a game is publicly available information and can be retrieved from EPG (electronic programming guide) metadata. Using the location of device101enables server108to discard content from devices not in the correct location and to avoid associating it with an event. For example, if a football game is being played at Levi's Stadium but a video was captured at a different location, then such video should not be associated with the broadcast of the game.

In another embodiment, server108polices content upload from devices101through N by discarding irrelevant frames based on the context. For example, server108might suspend devices that are contributing irrelevant content. Irrelevant content is any uploaded content that is not related to the context that the video is associated with. In some embodiments server108performs a pre-processing step to determine irrelevant content or frames and discard them (i.e., does not include them in further processing). This enables faster processing by processing only relevant content to generate an amalgamated recording. In one embodiment the system150might prioritize uploads from devices101with a high relevancy score. For example, scores may be assigned to devices101that previously uploaded desirable content (i.e., its previous uploads were popular). Prioritization can also be based on the device101capability (e.g., camera specifications) or proximity to an object of interest.

It should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and flowcharts or examples relating to one embodiment may be combined with any other embodiment in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.