Patent Description:
Automatic video content production of sport events has become more and more popular in recent years with the introduction of dedicated hardware and software. For many years, it has been suggested that advertisements can be fused into the video content in a manner that will enable viewers to watch the video broadcast with some portions of the surface exhibiting advertising content.

However, many challenges and specifically those relating to the relationship between foreground objects (e.g., players) and background surface prevented satisfactory results, mainly due to obstruction interference between the fused content and the foreground objects.

<CIT>, discloses a method for enabling control of insertions in a video. In the embodiment, control data is received from a user input device. Movement of at least one point of interest in a video is analysed to determine video metadata. Finally, a visual element is inserted into a video according to the control data, and the visual element changes or moves with the video as specified by the video metadata to appear integrated with the video.

<CIT>, discloses a method utilizing spatial modelling to represent foreground objects of an event to allow virtual graphics to be integrated into a background of the event in the presence of dynamic objects. The method includes detecting a presence of dynamic objects within a region of interest from a video depicting the event. The method further includes producing a suppression key corresponding to the dynamic object when present in the video or a suppression key with a default value when and where no dynamic object is present in the video.

One aspect of the present invention provides a method of fusing viewer-specific graphic content into a video content being broadcasted to a plurality of viewer terminals, the method comprising: receiving, by a video processing server, a video content comprising a plurality of frames captured by one or more cameras, wherein each of the plurality of frames represents a scene comprising background surface and objects of interest; determining, by the video processing server, physical parameters of each of the one or more cameras using at least one of: sensors located on the camera, computer vision methods and/or by panoramic capturing of the scene using plurality of cameras; deriving, by the video processing server, for each frame of a subset of frames of the plurality of frames, a virtual camera model that correlates each of pixel of the respective frame with a real-world geographic location in the scene associated with the pixel thereof, the virtual camera model being based on the physical parameters of the camera that generated the frame for which the respective virtual camera model has been derived; generating, by the video processing server, a background image, based on at least some of the plurality of frames of video content including pixels relating to the background surface of the scene, and subtracting, by the video processing server, the background image from each frame of the subset of frames of the video content to yield a corresponding subset of foreground masks, each foreground mask comprising pixels relating to the objects of interest; broadcasting, by the video processing server, the video content to at least some of the plurality of viewer terminals, wherein each frame of the subset of frames is accompanied with metadata that includes the respective frame's virtual camera model and foreground mask; and substituting, by at least some of the plurality of viewer terminals, in at least part of the frames of the subset of frames of the video content, all pixels in the respective frames contained within at least one predefined content insertion region of the background surface, except for the pixels indicated by the respective frames' foreground masks, with pixels of viewer-specific graphic contents associated with the viewer terminals thereof, using the respective frames' virtual camera models.

Another aspect of the present invention provides a system for fusing viewer-specific graphic content into a video content being broadcasted to a plurality of viewer terminals, the system comprising: a video processing server arranged to: receive a video content comprising a plurality of frames captured by one or more cameras, wherein each of the plurality of frames represents a scene comprising background surface and objects of interest; determine physical parameters of each of the one or more cameras using at least one of: sensors located on the camera, computer vision methods and/or by panoramic capturing of the scene using plurality of cameras; derive, for a subset of frames of the plurality of frames of the video content, a virtual camera model correlating each pixel of the respective frame with a real-world geographic location in the scene associated with the pixel thereof, the virtual camera model being based on the physical parameters of the camera that generated the frame for which the respective virtual camera model has been derived; generate a background image, based on at least some of the plurality of frames of video content including pixels relating to the background surface of the scene, and subtracting the background image from each frame of the subset of frames of the video content to yield a corresponding subset of foreground masks, each foreground mask comprising pixels relating to the objects of interest; and broadcast the video content to at least some of the plurality of viewer terminals, wherein each frame of the subset of frames is accompanied with metadata that includes the respective frame's virtual camera model and foreground mask; and a plurality of viewer terminals in communication with the video processing server, wherein said at least some of the plurality of viewer terminals is arranged to substitute, in at least part of the frames of the subset of frames, all pixels in the respective frames contained within a predefined content insertion region of the background surface, except for the pixels indicated by the respective foreground masks as related to the objects of interest, with pixels of viewer-specific graphic contents associated with the respective viewer terminals thereof, using the respective virtual camera models.

These, additional, and/or other aspects and/or advantages of the present invention are set forth in the detailed description which follows; possibly inferable from the detailed description; and/or learnable by practice of the present invention.

For a better understanding of embodiments of the invention and to show how the same can be carried into effect, reference will now be made, purely by way of example, to the accompanying drawings in which like numerals designate corresponding elements or sections throughout.

It will be appreciated that, for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale.

In the following description, various aspects of the present invention are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will also be apparent to one skilled in the art that the present invention can be practiced without the specific details presented herein. Furthermore, well known features can have been omitted or simplified in order not to obscure the present invention. With specific reference to the drawings, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the present invention only and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention can be embodied in practice.

Before at least one example of the invention is explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments that can be practiced or carried out in various ways as well as to combinations of the disclosed embodiments.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing", "computing", "calculating", "determining", "enhancing" or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulates and/or transforms data represented as physical, such as electronic, quantities within the computing system's registers and/or memories into other data similarly represented as physical quantities within the computing system's memories, registers or other such information storage, transmission or display devices. Any of the disclosed modules or units can be at least partially implemented by a computer processor.

Examples of the present invention provide a system and a method for fusing viewer-specific graphic content (such as advertisements) into a video content being broadcasted to a plurality of user terminals. The system may include a video processing server arranged to receive, or generate, a video content representing a scene at, for example, a sport event (e.g., such as soccer, basketball, football, etc.). The video content may include a plurality of frames.

The video processing server may derive, for a subset of frames of the plurality of frames of the video content, corresponding subset of virtual camera models and corresponding subset of foreground masks. In some embodiments, the frames of the subset are selected based on specified time period(s)/duration(s) during the sport event during which the viewer-specific graphic content is intended to be fused into the video content being broadcasted. The video processing server may further broadcast the video content in which the frames of the subset are accompanied with metadata that includes the respective virtual camera models and the respective foreground masks.

In various examples, the video content may be received at viewer terminals or at a virtual rendering server together with viewer-specific graphic content (such as advertisements, logos, etc.) to be fused therein.

The viewer terminals / the virtual rendering server may be arranged to fuse the user-specific graphic content into the video content by substituting all pixels in the frames of the subset that are contained within a predefined content insertion region in the scene, except for the pixels that are indicated by the respective frames' foreground masks, and using the respective frames' virtual camera models, with pixels of the user-specific graphic content thereof.

In various examples, the viewer-specific graphic content may be tailored to each of the viewers individually or to different groups of viewers (e.g., men, women, children, etc.). Thus, in some examples, each of the viewer terminals, or groups of the viewer terminals, may receive video content with the viewer-tailored and possibly different graphic content being fused therein, wherein the fusion takes into account the foreground so as to eliminate collisions and obstructions.

Advantageously, the disclosed system and method may enable fusion of alternative graphic content into the video content being broadcasted directly on either viewer terminals or virtual rendering server (and remotely from the video processing server), thereby providing high flexibility in tailoring the alternative graphic content to specific viewers / groups of viewers while eliminating a need in repeatable, complex and resource consuming preparation stages of the video content (e.g., generation of the foreground masks and virtual camera models) that may be performed only once on the video processing server.

Reference is now made to <FIG>, <FIG> and <FIG>, which are schematic illustrations of various configurations of a system <NUM> for fusing viewer-specific graphic content into a video content being broadcasted to a plurality of user terminals, according to some examples of the invention.

According to some examples, system <NUM> may include a video processing server <NUM> and a plurality of user terminals <NUM>(<NUM>) to <NUM>(M) (such as, for example, smartphones, tablet computers, clouds, smart TVs, etc.) in communication with video processing server <NUM>. In some embodiments, user terminals <NUM>(<NUM>) to <NUM>(M) may be associated with a plurality of viewers <NUM>(<NUM>) to <NUM>(M).

Video processing server <NUM> may receive (e.g., locally or over a network) a video content <NUM> (e.g., as shown in <FIG>). Video content <NUM> may include a plurality of frames. Video content <NUM> may represent a scene <NUM> at, for example, a sport event (e.g., soccer game, basketball game, etc.). Scene <NUM> may, for example, include a stationary/background surface <NUM> and objects of interest <NUM>. For example, stationary/background surface <NUM> may be a game-field and/or objects of interest <NUM> may be moving objects such as a ball <NUM>, players 24a to 24e and/or a referee <NUM> of the sport event. Accordingly, the frames of video content <NUM> may include pixels relating to stationary/background surface <NUM> and pixels relating to objects of interest <NUM>.

System <NUM> includes at least one camera <NUM> (e.g., static or dynamic camera). Camera <NUM> may be directed at, for example, scene <NUM> of the sport event and may be arranged to capture video footage and convey its respective video stream <NUM> to video processing server <NUM>. In these embodiments, video processing server <NUM> may be arranged to receive video stream <NUM> and to generate video content <NUM> based on video stream <NUM> (e.g., as described below with respect to <FIG> and <FIG>).

In some embodiments, system <NUM> may include plurality of cameras <NUM>(<NUM>) to <NUM>(N) directed at scene <NUM> of the sport event arranged to capture video footages and to convey their respective video streams <NUM>(<NUM>) to <NUM>(N) to video processing server <NUM> (e.g., as shown in <FIG> and <FIG>). Video processing server <NUM> may be further arranged to generate video content <NUM> based on at least some of video streams <NUM>(<NUM>) to <NUM>(N). In these embodiments, each of plurality of cameras <NUM>(<NUM>) to <NUM>(N) may be directed at a different angle so as all of plurality of cameras <NUM>(<NUM>) to <NUM>(N) together may provide a panoramic view of the scene and video content <NUM> (being generated by video processing server <NUM>) may further include all possible angles the scene <NUM>.

Video processing server <NUM> is arranged to derive, for each frame of a subset of frames of the plurality of frames of video content <NUM>, a virtual camera model, to yield corresponding subset of virtual camera models <NUM> (e.g., as described below with respect to <FIG>). The virtual camera model of each frame of the subset of frames of video content <NUM> may, for example, correlate each of the pixels of the respective frame with a real-world geographic location associated with the pixel thereof (e.g., as described below with respect to <FIG>).

In some embodiments, the frames of the subset are selected based on specified time period(s)/duration(s) of the sport event during which the viewer-specific graphic content is intended to be fused into the video content. In some embodiments, the subset of frames includes all the plurality of frames of video content <NUM>.

Video processing server <NUM> is arranged to generate, for each frame of the subset of frames of video content <NUM>, a foreground mask, to yield corresponding subset of foreground masks <NUM> (e.g., as described below with respect to <FIG> and <FIG>). The foreground mask of each frame of the subset of frames may, for example, include pixels relating to objects of interest <NUM> (ball <NUM>, players <NUM>, referee of the sport event <NUM>, etc.).

Video processing server <NUM> is arranged to broadcast (for example, over a network) video content <NUM> to at least some of plurality of user terminals <NUM>(<NUM>) to <NUM>(M), wherein each frame of the subset of frames of video content <NUM> is accompanied with metadata that includes the respective frame's virtual camera model and the respective frame's foreground mask (e.g., as shown in <FIG> and <FIG>).

At least some of user terminal <NUM>(<NUM>) to <NUM>(M) may receive a respective and possibly different viewer-specific graphical content <NUM>(<NUM>) to <NUM>(M) (such as advertisement, logos, etc.) (e.g., as shown in <FIG> and <FIG>). In various embodiments, each (or at least some) of viewer-specific graphic content <NUM>(<NUM>) to <NUM>(M) may be tailored to each of the viewers <NUM>(<NUM>) to <NUM>(M) individually or to different groups of viewers (e.g., men, women, children, etc.).

At least some of user terminals <NUM>(<NUM>) to <NUM>(M) may be arranged to substitute (e.g., by a computer processor) all pixels in at least part of the frames of the subset of frames of video content <NUM> that are contained within a predefined content insertion region <NUM> of background/stationary surface <NUM>, except for the pixels that are indicated by the respective frames' foreground masks <NUM> as relating to objects of interest <NUM> (e.g., to players 24c, 24d, as shown in <FIG> and <FIG>), with pixels of user-specific graphic content <NUM>(<NUM>) to <NUM>(M) associated with the at least some user terminals thereof, using the respective frames' virtual camera models <NUM>. Thus, at least some of viewer terminals <NUM>(<NUM>) to <NUM>(M) may receive locally a different content to be fused on predefined content insertion region <NUM>, wherein the substitutions of the pixels with the fused content takes into account objects of interest <NUM> (e.g., moving objects) of the foreground so as to eliminate collisions and obstructions.

According to some examples, system <NUM> may include a virtual rendering server <NUM> (e.g., as shown in <FIG>). Virtual rendering server <NUM> may be in communication with video processing server <NUM> and with user terminals <NUM>(<NUM>) to <NUM>(M). Virtual rendering server <NUM> may receive from video processing server <NUM> video content <NUM>, wherein each frame of the subset of frames of video content <NUM> is accompanied with the metadata (e.g., described above with respect to <FIG> and <FIG>).

Virtual rendering server <NUM> may be further arranged to receive viewer-specific graphic content <NUM>(<NUM>) to <NUM>(M). In some embodiments, at least some pixels of at least some of viewer-specific graphic content <NUM>(<NUM>) to <NUM>(M) may have a predetermined measure of transparency.

Virtual rendering server <NUM> may be arranged to generate user-specific video contents <NUM>(<NUM>) to <NUM>(M) by substituting all pixels in at least part of the frames of the subset of frames of video content <NUM> that are contained within predefined content insertion region <NUM> of background/stationary surface <NUM>, except for the pixels that are indicated by the respective frames' foreground masks <NUM> as relating to objects of interest <NUM>, with pixels of the corresponding user-specific graphic contents <NUM>(<NUM>) to <NUM>(M), using the respective frames' virtual camera models <NUM>. Virtual rendering server <NUM> may be further arranged to broadcast at least some of user-specific video contents <NUM>(<NUM>) to <NUM>(M) to at least some of user terminal(s) <NUM>(<NUM>) to <NUM>(M).

Reference is now made to <FIG>, <FIG> and <FIG>, which are schematic illustration of various configurations of a more detailed aspect of a system <NUM> for fusing viewer-specific content into a video production, according to some examples of the invention.

According to some embodiments, system <NUM> may include a video processing server <NUM> and a plurality of user terminals <NUM> in communication with video server <NUM> (e.g., as shown in <FIG> and <FIG>).

According to some examples, video processing server <NUM> may receive a video content <NUM> (e.g., as shown in <FIG>). Video content <NUM> may include a plurality of frames each representing scene <NUM> at the sport event (e.g., as described above with respect to <FIG>, <FIG> and <FIG>).

According to some examples, video processing server <NUM> may include a video production generator <NUM> (e.g., as shown in <FIG> and <FIG>). Video production generator <NUM> may, for example, receive plurality of video streams <NUM>(<NUM>) to <NUM>(N) (e.g., generated by corresponding plurality of video cameras, such as cameras <NUM>(<NUM>) to <NUM>(N) directed at scene <NUM> of the sport event, as described above with respect to <FIG>). Video production generator <NUM> may generate a video content <NUM> including a plurality of frames, based on video streams <NUM>(<NUM>) to <NUM>(N).

For example, video production module <NUM> may selectively combine video streams <NUM>(<NUM>) to <NUM>(N) through video editing into video content <NUM> to "tell a story" of the sport event. The video editing may, for example, include creating combinations and/or reductions of parts of video streams <NUM>(<NUM>) to <NUM>(N) in either live event setting (e.g., live production) or after the sport event has occurred (e.g., post-production).

Video processing server <NUM> includes a foreground mask generator <NUM>. Foreground mask generator <NUM> may be arranged to generate a foreground mask for each frame of a subset <NUM> of frames of the plurality of frames of video content <NUM>, to yield corresponding subset of foreground masks <NUM>. For example, each of plurality of foreground masks <NUM> may be generated for one frame of subset <NUM>. Each of plurality of foreground masks <NUM> may include pixels relating to objects of interest <NUM> in scene <NUM> (e.g., as described above with respect to <FIG>, <FIG> and <FIG>). In some embodiments, the frames of subset <NUM> are selected based on specified time period(s)/duration(s) of the sport event during which the viewer-specific graphic content is intended to be fused into the video content.

Foreground mask generator <NUM> may utilize background subtraction methods to generate foreground masks <NUM>. Foreground mask generator <NUM> may determine a background image, based on at least some of plurality of frames of video content <NUM>. The background image may, for example, include pixels relating to stationary/background surface <NUM> of the scene. Foreground mask generator <NUM> may, for example, subtract the background image (that includes pixels relating to background/stationary surface <NUM>) from each frame of the subset of frames of video content <NUM> (that include pixels relating to both background/stationary surface <NUM> and objects of interest <NUM>) to yield corresponding subset of foreground masks images <NUM> (that include pixels relating to objects of interest <NUM>). It would be appreciated that foreground mask generator <NUM> may also utilize other background subtraction techniques.

Foreground mask generator <NUM> may utilize chromakeying methods to generate foreground masks <NUM>. Foreground mask generator <NUM> may, for example, detect and remove all pixels in the frames of the subset of frames of video content <NUM> relating to background/stationary surface <NUM> (e.g., that may have substantially the same color) to thereby generate corresponding subset of foreground masks images <NUM> (that include pixels relating to objects of interest <NUM>). In these embodiments, foreground masks <NUM> may further include elements on background/stationary surface <NUM> whose color differs from a dominant color of background/stationary surface <NUM> (e.g., white-line markings, etc.).

It would be appreciated that foreground mask generator <NUM> may utilize other methods (e.g., rather than background subtraction and/or chromakeying) to generate foreground masks <NUM>, for example deep learning algorithms.

According to some examples, system <NUM> may include virtual camera model generator <NUM>. Virtual camera generation model <NUM> may derive, for each frame of subset <NUM> of frames of video content <NUM>, a virtual camera model, to yield corresponding subset of virtual camera models <NUM>. For example, each of plurality of virtual camera models <NUM> may be derived for one frame of subset <NUM>.

Each of virtual camera models <NUM>, derived for one frame of subset <NUM> of video content <NUM>, may correlate each pixel in the respective frame with a real-world geographic location in scene <NUM> associated with the pixel thereof. The correlation thereof may be done based on, for example, physical parameters of a camera that generated the respective frame. The physical parameters thereof may, for example, include at least a real-world geographic location of the camera with respect to scene <NUM>, orientation of the camera with respect to scene <NUM> and/or lens parameters such as focal length, distortion, etc..

The physical parameters of the camera(s) may be determined using, for example, at least one of: sensors located on the camera(s), computer vision methods and/or by panoramic capturing of the scene using plurality of cameras (e.g., such as cameras <NUM>(<NUM>) to <NUM>(N), as described above with respect to <FIG>). Alternatively or complementarily, the physical parameters attributed to the camera(s) thereof may be received by virtual camera generation model <NUM> as metadata.

In some embodiments, video processing server <NUM> may receive as an input at least one content insertion region <NUM>. Content insertion region(s) <NUM> may include information concerning real-world geographic location(s) on background/stationary surface <NUM> of scene <NUM> to be replaced with the viewer-specific content (e.g., as described above with respect to <FIG> and <FIG>). Content insertion region(s) <NUM> may be, for example, on a game-field, area(s) surrounding a stadium at which the sport event takes place and/or the entire scene <NUM>. In various embodiments, at least some frames of subset <NUM> may include single content insertion region <NUM>, or two or more content insertion regions <NUM>.

Video processing server <NUM> may generate metadata <NUM> for the frames of subset <NUM>, meaning that each frame of subset <NUM> may be accompanied with the respective frame's foreground mask <NUM>, the respective mask virtual camera model <NUM> and content insertion region(s) <NUM> (e.g., as shown in <FIG>, <FIG> and <FIG>). In some embodiments, video processing server <NUM> may further broadcast video content <NUM>, wherein each frame of subset <NUM> is accompanied with the respective frame's metadata <NUM>.

According to some examples, video content <NUM> and metadata <NUM> being broadcasted by video production server <NUM> may be received by at least some of viewer terminals <NUM>. Each of at least some of viewer terminals <NUM> may further receive as an input a respective and possibly different viewer-specific graphical content <NUM> (e.g., as shown in <FIG> and <FIG>). In some examples, at least some pixels of at least some of viewer-specific graphic content <NUM> may have a predetermined measure of transparency.

At least some of user terminals <NUM> may include a virtual rendering module <NUM>. Virtual rendering module <NUM> of each of viewer terminals <NUM> may be arranged to substitute, for at least part of frames of subset <NUM> of video content <NUM>, using the respective frames' virtual camera models, all pixels that are contained within predefined content insertion region(s) <NUM>, except for the pixels that are indicated by the respective frames' foreground masks as relating to objects of interest <NUM>, with pixels of user-specific graphic content <NUM> associated with the respective user terminal.

Thus, at least some of viewers associated with viewer terminals <NUM> may receive locally a different content to be fused on predefined content insertion region(s) <NUM>, wherein the substitutions of the pixels with the fused content takes into account objects of interest <NUM> (e.g., moving objects) of the foreground so as to eliminate collisions and obstructions.

According to some examples, system <NUM> may include a virtual rendering server <NUM> (e.g., as shown in <FIG>). Virtual rendering server <NUM> may be in communication with video processing server <NUM> and with user terminals <NUM>. Virtual rendering server <NUM> may receive from video processing server <NUM> video content <NUM>, wherein each frame of subset <NUM> is accompanied with metadata <NUM> (e.g., described above with respect to <FIG> and <FIG>). Virtual rendering server <NUM> may be further arranged to receive viewer-specific graphic contents <NUM> (e.g., as shown in <FIG>). In some embodiments, viewer-specific graphic contents <NUM> may include plurality of alternative graphic contents tailored to specific viewers / groups of viewers (e.g., as described above with respect to <FIG>).

Virtual rendering server <NUM> may be arranged to generate user-specific video contents <NUM> by substituting all pixels in at least part of the frames of subset <NUM> of frames of video content <NUM> that are contained within predefined content insertion region(s) <NUM> of background/stationary surface <NUM>, except for the pixels that are indicated by the respective frames' foreground masks <NUM> as relating to objects of interest <NUM>, with pixels of the corresponding user-specific graphic contents <NUM>, using the respective frames' virtual camera models <NUM>. Virtual rendering server <NUM> may be further arranged to broadcast at least some of user-specific video contents <NUM> to at least some of user terminals <NUM> (e.g., as shown in <FIG>).

Reference is now made to <FIG>, which is a flowchart illustrating a method of fusing viewer-specific content into a video content being broadcasted to a plurality of viewer terminals, according to some embodiments of the invention.

According to some examples, the method may be implemented by system <NUM> or system <NUM>, which may be configured to implement the method. It is noted that the method is not limited to the flowcharts illustrated in <FIG> and to the corresponding description. For example, in various examples, the method needs not move through each illustrated box or stage, or in exactly the same order as illustrated and described.

In some examples, the method may include receiving, by a video processing server, a video content including a plurality of frames, wherein each of the plurality of frames represents a scene (e.g., of a sport event) and includes pixels relating to a stationary/background surface in the scene and pixels relating to objects of interest in the scene (stage <NUM>).

The method may include receiving, by the video processing server, at least one video stream (e.g., from at least one camera directed at the scene), and further generating the video content based on the at least one video stream (stage <NUM>).

The method may include selectively creating combinations and/or reductions of parts of the at least one video stream in either live event setting (e.g., live production) or after the sport event has occurred (e.g., post-production) to thereby generate the video content (stage <NUM>).

The method may include deriving, for each frame of a subset of frames of the plurality of frames, a virtual camera model that correlates each of pixels of the respective frame with a real-world geographic location in the scene associated with the pixel thereof, to yield corresponding subset of virtual camera models (stage <NUM>).

The method may include deriving each of the subset of virtual camera models based on physical parameters of a camera that generated the frame for which the respective virtual camera model has been derived (e.g., at least a real-world geographic location of the camera with respect to the scene, orientation of the camera with respect to the scene and/or les parameters such as focal length, distortion, etc.) (stage <NUM>).

The method may include determining, by the video processing server, the physical parameters of the respective camera using at least one of sensors located on the camera, computer vision methods and/or by panoramic capturing of the scene using plurality of cameras (stage <NUM>).

The method may include generating, by the video processing server, for each frame of the subset of frames of video content <NUM>, a foreground mask comprising the pixels relating to the objects of interest, to yield corresponding subset of foreground masks (stage <NUM>).

The method may include generating a background image, based on at least some of the plurality of frames of video content including pixels relating to the background surface of the scene (e.g., as described above with respect to <FIG>) (stage <NUM>).

The method may include subtracting the background image from each frame of the subset of frames of the video content to yield the corresponding subset of foreground masks (stage <NUM>).

The method may include detecting and removing all pixels in the frames of the subset of frames of the video content relating to the background surface to thereby generate the corresponding subset of foreground masks images (stage <NUM>).

The method may include receiving as an input at least one content insertion region including information concerning a real-world geographic location on the background surface of the scene to be replaced with a viewer-specific content (stage <NUM>).

The method may include generating, by the video processing server, for each frame of the subset of frames of the video content, metadata including the respective frame's foreground mask and the respective frame's virtual camera model (stage <NUM>).

The method may include broadcasting, by the video processing server, the video content with the metadata (stage <NUM>).

The method may further include receiving, by at least some of the plurality of viewer terminals, the video content with the metadata and viewer-specific graphic content (stage <NUM>).

The method may include substituting, by the at least some of the plurality of viewer terminals, in at least part of the frames of the subset of frames of the video content, all pixels in the respective frames contained within the at least one content insertion region of the background surface, except for the pixels indicated by the respective frames' foreground masks, with pixels of the viewer-specific graphic content, using the respective frames' virtual camera models (stage <NUM>).

The method may further include receiving, by a virtual rendering server, the video content with the metadata and viewer-specific graphic content (stage <NUM>).

The method may include substituting, by the virtual rendering server, in at least part of the frames of the subset of frames of the video content, all pixels in the respective frames contained within the at least one content insertion region of the background surface, except for the pixels indicated by the respective frames' foreground masks, with pixels of the viewer-specific graphic content, using the respective frames' virtual camera models, to thereby generate viewer-specific video content (stage <NUM>).

The method may further include broadcasting the viewer-specific video content to at least some of the plurality of viewer terminals (stage <NUM>).

The method may further include tailoring the viewer-specific graphic content to specific viewers or to specific groups of viewers such that at least some of the plurality of viewer terminals are provide with different viewer-specific graphic content (stage <NUM>).

Aspects of the present invention are described above with reference to flowchart illustrations and/or portion diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each portion of the flowchart illustrations and/or portion diagrams, and combinations of portions in the flowchart illustrations and/or portion diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or portion diagram or portions thereof.

These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or portion diagram portion or portions thereof. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or portion diagram portion or portions thereof.

The aforementioned flowchart and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each portion in the flowchart or portion diagrams can represent a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the portion can occur out of the order noted in the figures. For example, two portions shown in succession can, in fact, be executed substantially concurrently, or the portions can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each portion of the portion diagrams and/or flowchart illustration, and combinations of portions in the portion diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In the above description, an embodiment is an example or implementation of the invention. The various appearances of "one embodiment", "an embodiment", "certain embodiments" or "some embodiments" do not necessarily all refer to the same embodiments. Although various features of the invention can be described in the context of a single embodiment, the features can also be provided separately or in any suitable combination. Conversely, although the invention can be described herein in the context of separate embodiments for clarity, the invention can also be implemented in a single embodiment. Certain embodiments of the invention can include features from different embodiments disclosed above, and certain embodiments can incorporate elements from other embodiments disclosed above. The disclosure of elements of the invention in the context of a specific embodiment is not to be taken as limiting their use in the specific embodiment alone. Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in certain embodiments other than the ones outlined in the description above.

Claim 1:
A method of fusing viewer-specific graphic content into a video content being broadcasted to a plurality of viewer terminals, the method comprising:
receiving, by a video processing server, a video content comprising a plurality of frames captured by one or more cameras, wherein each of the plurality of frames represents a scene comprising background surface and objects of interest (<NUM>);
determining, by the video processing server, physical parameters of each of the one or more cameras using at least one of: sensors located on the camera, computer vision methods and/or by panoramic capturing of the scene using plurality of cameras (<NUM>);
deriving, by the video processing server, for each frame of a subset of frames of the plurality of frames, a virtual camera model that correlates each pixel of the respective frame with a real-world geographic location in the scene associated with the pixel thereof (<NUM>), the virtual camera model being based on the physical parameters of the camera that generated the frame for which the respective virtual cameramodel has been derived (<NUM>);
generating, by the video processing server, a background image, based on at least some of the plurality of frames of video content including pixels relating to the background surface of the scene (<NUM>), and subtracting, by the video processing server, the background image from each frame of the subset of frames of the video content to yield a corresponding subset of foreground masks, each foreground mask comprising pixels relating to the objects of interest (<NUM>);
broadcasting, by the video processing server, the video content to at least some of the plurality of viewer terminals, wherein each frame of the subset of frames is accompanied with metadata that includes the respective frame's virtual camera model and foreground mask (<NUM>); and
substituting, by said at least some of the plurality of viewer terminals, in at least part of the frames of the subset of frames of the video content, all pixels in the respective frames contained within at least one predefined content insertion region of the background surface, except for the pixels indicated by the respective frames' foreground masks, with pixels of viewer-specific graphic contents associated with the viewer terminals thereof, using the respective frames' virtual camera models (<NUM>).