Patent ID: 12190914

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

It is to be understood that the present disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The present disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

The use of “including”, “comprising” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms “a” and “an” herein do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. Further, the use of terms “first”, “second”, and “third”, and so forth, herein do not denote any order, quantity, or importance, but rather are used to distinguish one element from another.

Referring toFIG.1is a block diagram100depicting a schematic representation of a system for extracting objects from videos in real-time to create virtual situations, in accordance with one or more exemplary embodiments. The system100includes a computing device102, a network104, a server106, a processor108, a camera110, a memory112, a video creating and editing module114, a video processing module116, a database server118, and a database120.

The computing device102may include the user's devices. The computing device102may include, but is not limited to, a personal digital assistant, smartphones, personal computers, a mobile station, computing tablets, a handheld device, an internet enabled calling device, an internet enabled calling software, a telephone, a mobile phone, a digital processing system, and so forth. The computing device102may include the processor108in communication with a memory112. The processor108may be a central processing unit. The memory112is a combination of flash memory and random-access memory.

The computing device102may be communicatively connected with the server106via the network104. The network104may include, but not limited to, an Internet of things (IoT network devices), an Ethernet, a wireless local area network (WLAN), or a wide area network (WAN), a Bluetooth low energy network, a ZigBee network, a WIFI communication network e.g., the wireless high speed internet, or a combination of networks, a cellular service such as a 4G (e.g., LTE, mobile WiMAX) or 5G cellular data service, a RFID module, a NFC module, wired cables, such as the world-wide-web based Internet, or other types of networks may include Transport Control Protocol/Internet Protocol (TCP/IP) or device addresses (e.g. network-based MAC addresses, or those provided in a proprietary networking protocol, such as Modbus TCP, or by using appropriate data feeds to obtain data from various web services, including retrieving XML data from an HTTP address, then traversing the XML for a particular node) and so forth without limiting the scope of the present disclosure.

Although the computing device102is shown inFIG.1, an embodiment of the system100may support any number of computing devices. The computing device102may be operated by the user. The user may include, but not limited to, an individual, a client, an operator, a content creator, and the like. The computing device102supported by the system100is realized as a computer-implemented or computer-based device having the hardware or firmware, software, and/or processing logic needed to carry out the computer-implemented methodologies described in more detail herein.

In accordance with one or more exemplary embodiments of the present disclosure, the computing device102may include the camera110may be configured to enable the user to capture the multimedia objects using the processor108. The multimedia objects may include, but not limited to photos, snaps, short videos, videos, and the like. The computing device102may include the video creating and editing module114in the memory112.

The video creating and editing module114may be configured to enable the user to create or record videos or upload pre-recorded videos or photos on the computing device102. The video creating and editing module114may be any suitable applications downloaded from GOOGLE PLAY® (for Google Android devices), Apple Inc.'s APP STORE® (for Apple devices), or any other suitable database. The video creating and editing module114may be a desktop application which runs on Windows or Linux or any other operating system and may be downloaded from a webpage or a CD/USB stick etc. In some embodiments, the video creating and editing module114may be software, firmware, or hardware that is integrated into the computing device102. The computing devices102may present a web page to the user by way of a browser, wherein the webpage comprises a hyper-link may direct the user to uniform resource locator (URL).

The server106may include the video processing module116, the database server118, and the database120. The video processing module116may be configured to detect the objects from the user created or recorded videos. The video processing module116may be configured to automatically extract the detected objects from the user created or recorded videos. The objects may include but not limited to humans, the human body, animals, any other objects, and the like. The video processing module116may also be configured to provide server-side functionality via the network104to one or more users. The database server118may be configured to access the one or more databases. The database120may be configured to store user created and recorded videos. The database120may also be configured to store interactions between the modules of the video creating and editing module114, and the video processing module116.

In accordance with one or more exemplary embodiments of the present disclosure, the computing device102may be configured to establish communication with the server106over the network104. The computing device102may include the video creating and editing module114. The video creating and editing module114may be configured to enable the user to record one or more videos. The video creating and editing module114may be configured to select one or more frames automatically from the one or more user recorded videos. The video creating and editing module114may be configured to transfer the one or more automatically selected frames from the computing device102to the server106over the network104. The server104may include the video processing module116configured to receive the one or more automatically selected frames. The video processing module116may be configured to detect and extract one or more objects from the one or more automatically selected frames and transfer the one or more extracted objects to the computing device102over the network104. The video creating and editing module114may be configured to receive one or more extracted objects and display the one or more extracted objects to the user on the computing device102. The video creating and editing module114may be configured to place the one or more extracted objects on the new frame automatically. The video creating and editing module114may be configured to allow the user to reposition the one or more extracted objects on the new frame. The video creating and editing module114may be configured to enable the user to customize background and foreground elements in the new frame to create virtual situations.

Referring toFIG.2is a block diagram200depicting an embodiment of the video creating and editing module114on the computing device and the video processing module116on the server of shown inFIG.1, in accordance with one or more exemplary embodiments. The video creating and editing module114includes a bus201a, a registration module202, an authentication module204, a video recording module206, a frame selection module208, an objects displaying module210, an objects selection module212, a new frame selection module214, an objects placing module216, a background and foreground customization module218, and a video posting module220. The bus201amay include a path that permits communication among the modules of the video creating and editing module114installed on the computing device102. The term “module” is used broadly herein and refers generally to a program resident in the memory112of the computing device102.

The registration module202may be configured to enable the user to register on the video creating and editing module114installed on the computing device102by providing basic details of the user. The basic details may include but not limited to email, password, first and last name, phone number, address details, and the like. The registration module202may also be configured to transfer the user registration details to the server106over the network104. The server114may include the video processing module116. The video processing module116may be configured to receive the user registration details from the registration module202. The authentication module204may be configured to enable the user to log in and access the video creating and editing module114installed on the computing device102by using the user login identity credentials. The video recording module206may be configured to enable the user to tap a camera icon to record the video. The video recording module206may also be configured to enable the user to upload pre-recorded videos on the computing device102. The frame selection module208may be configured to display a series of frames to the user from the user recorded video. The frame selection module208may also be configured to enable the user to select a frame from the series of frames. The frame selection module208may also be configured to transfer the user selected frame to the server106through the network104. The frame selection module208may also be configured to enable the user to select a new frame from the user recorded one or more videos. The objects displaying module210may be configured to receive the extracted objects from the video processing module116. The objects displaying module210may also be configured to display the extracted objects to the user. The objects selection module212may be configured to enable the user to select the extracted objects to transfer into the new frame to create a virtual situation. The new frame selection module214may be configured to enable the user to select a new frame from the user created video to create new virtual situations in the video. The objects placing module216may be configured to enable the user to place the selected objects into the new frame. The background and foreground customization module218may be configured to enable the user to customize the background and foreground elements in the new frame. The background and foreground customization module218may be configured to enable the user to upload their own assets while customizing the background and foreground elements. The video posting module220may be configured to enable the user to post the recorded video on the computing device102. The video posting module220may also be configured to enable the user to post the recorded video with newly created virtual situations in the video. The video posting module220may also be configured to enable the user to preview the video with newly created virtual situations before posting on the computing device102.

In accordance with one or more exemplary embodiments of the present disclosure, the video processing module116includes a bus201b, an authentication data processing module222, a frames receiving module224, and an objects extraction and classification module226. The bus201bmay include a path that permits communication among the modules of the video processing module116installed on the sever106.

The authentication data processing module222may be configured to receive the user registration details from the registration module202. The authentication data processing module222may also be configured to generate the user login identity credentials using the user registration details. The identity credentials comprise a unique identifier (e.g., a username, an email address, a date of birth, a house address, a mobile number, and the like), and a secured code (e.g., a password, a symmetric encryption key, biometric values, a passphrase, and the like). The frames receiving module224may be configured to receive the user selected frame from the frame selection module208. The objects extraction and classification module226may be configured to detect and extract the objects from the user selected frame. The objects extraction and classification module226may also be configured to classify the extracted objects into the appropriate categories. The objects extraction and classification module226may also be configured to transfer the extracted objects to the video creating and editing module114through the network104. The objects extraction and classification module226may be configured to extract the same objects from multiple frames of the video to create a new situation that has motion in it. The extracted objects may be shrunk or magnified to create out-of-proportion experiences. The objects extraction and classification module226may be configured to create duplicate situations and stories. For example, a human may be duplicated and mirrored to appear as if one is looking at oneself in a new situation. The objects extraction and classification module226may also be configured to guide the user selected frame by inflection points in a background audio track used in the video. Thereby the objects extraction and classification module226allows a visual transition that corresponds to an inflection in sound. The objects extraction and classification module226may also be configured to guide the user selected frame by visual changes in the video itself, for example, a pause after a dance move.

In accordance with one or more exemplary embodiments of the present disclosure, one or more videos may be automatically or manually mixed together before or after object extraction to bring new situations that involve multiple objects that are present in separate videos. Using this method, two or more people who have appeared in separate videos may come together in the new situation created in the combined video. Examples of such virtual situations include things like two or more people walking the streets of Paris together, a human appearing next to an animal, and many more.

In accordance with one or more exemplary embodiments of the present disclosure, the video creating and editing module114may be configured to select the one or more frames automatically from the one or more user recorded videos to create virtual situations on the computing device102without the cloud server. In accordance with one or more exemplary embodiments of the present disclosure, the extracted objects may include a human body, animals, and other objects in real-time from one or more frames of the video. In accordance with one or more exemplary embodiments of the present disclosure, the new frame may include a movie poster, a movie scene, a movie song ending, movie trailers, video memes, brand interactions, and virtual landscapes.

Referring toFIG.3is a block diagram300depicting the system for extracting objects from videos in real-time to create virtual situations, in accordance with one or more exemplary embodiments. The video creating and editing module114may be configured to enable the user to log in or signup to the video creating and editing module on the computing device by providing user details. The video creating and editing module114may also be configured to send the request to check the user details from the computing device to the server. The video processing module116may be configured to receive the request to check the user details. The video processing module116may also be configured to determine whether the user is an existing user or not. The video processing module116may also be configured to enable the existing user directly to access the video creating and editing module114to create the virtual situations. The video processing module116may also be configured to enable the new user to watch a tutorial for creating virtual situations.

Referring toFIG.4are example screens400depicting a video creating and editing module, in accordance with one or more exemplary embodiments. The screens400includes multimedia screens402a,402b,402c,402d,402e,402f,402g,402h,402,402i,402j,402k,402l,402m,402n, and402o. The screens402a,402b,402c,402d,402e,402f,402g,402h,402,402i,402j,402k,402l,402m,402n, and402odepicts new virtual situations in one or more videos. video creating and editing module configured to enable a user to record one or more videos. The video creating and editing module may be configured to display a series of frames to the user from one or more user recorded videos and allow the user to select one or more frames from the series of frames, the video creating and editing module configured to transfer the one or more user selected frames from the computing device to a server over a network. The server comprises a video processing module configured to receive one or more user selected frames. The video processing module may be configured to detect and extract one or more objects from one or more user selected frames and transfer one or more extracted objects to the computing device over the network. The video creating and editing module may be configured to receive one or more extracted objects and display the one or more extracted objects to the user on the computing device, the video creating and editing module may be configured to enable the user to select a new frame from the one or more user recorded videos and allow the user to select and place the one or more extracted objects into the new frame, the video creating and editing module configured to enable the user to customize background and foreground elements in the new frame, the video creating and editing module configured to delete the one or more user selected frames after extracting the one or more objects and replace one or more user selected frames with the new frame.

Referring toFIG.5is a flow diagram500depicting a method for extracting objects from videos in real-time to create virtual situations, in accordance with one or more exemplary embodiments. The method500may be carried out in the context of the details ofFIG.1,FIG.2,FIG.3, andFIG.4. However, the method500may also be carried out in any desired environment. Further, the aforementioned definitions may equally apply to the description below.

The method commences at step502, enabling the user to record one or more videos by the video creating and editing module on the computing device. Thereafter at step504, selecting one or more frames automatically from the one or more user recorded videos by the video creating and editing module. Thereafter at step506, transferring the one or more automatically selected frames from the computing device to the server by the video creating and editing module over a network. Thereafter at step508, receiving the one or more automatically selected frames by the video processing module enabled in the server. Thereafter at step510, detecting and extracting one or more objects from the one or more automatically selected frames by the video processing module. Thereafter at step512, transferring the one or more extracted objects to the computing device by the video processing module over the network. Thereafter at step514, receiving the one or more extracted objects by the video creating and editing module and displaying the one or more extracted objects to the user on the computing device. Thereafter at step516, placing the one or more extracted objects on the new frame automatically by the video creating and editing module. Thereafter at step518, allowing the user to reposition the one or more extracted objects on the new frame by the video creating and editing module. Thereafter at step520, enabling the user to customize background and foreground elements in the new frame to create virtual situations by the video creating and editing module.

Referring toFIG.6is a flowchart600depicting an exemplary method for enabling a user to access the video creating and editing module to create the virtual situations, in accordance with one or more exemplary embodiments. The method600may be carried out in the context of the details ofFIG.1,FIG.2,FIG.3,FIG.4, andFIG.5. However, the method600may also be carried out in any desired environment. Further, the aforementioned definitions may equally apply to the description below.

The method commences at step602, enabling the user to log in or signup to a video creating and editing module on a computing device by providing user details. Thereafter at step604, sending the request to check the user details from the computing device to the server by the video creating and editing module. Thereafter at step606, receiving the request to check the user details by the video processing module enabled in the server. Determining whether the user is an existing user or not, at step608. If answer at step608is Yes, the method continues at step610, enabling the user to access the video creating and editing module to create the virtual situations in the video. If answer at step608is No, the method continues at step612, displaying an option for the user to watch the tutorial for creating virtual situations in the video.

Referring toFIG.7is a block diagram700illustrating the details of a digital processing system700in which various aspects of the present disclosure are operative by execution of appropriate software instructions. The Digital processing system700may correspond to the computing device102(or any other system in which the various features disclosed above can be implemented).

Digital processing system700may contain one or more processors such as a central processing unit (CPU)710, random access memory (RAM)720, secondary memory730, graphics controller760, display unit770, network interface780, and input interface790. All the components except display unit770may communicate with each other over communication path750, which may contain several buses as is well known in the relevant arts. The components ofFIG.7are described below in further detail.

CPU710may execute instructions stored in RAM720to provide several features of the present disclosure. CPU710may contain multiple processing units, with each processing unit potentially being designed for a specific task. Alternatively, CPU710may contain only a single general-purpose processing unit.

RAM720may receive instructions from secondary memory730using communication path750. RAM720is shown currently containing software instructions, such as those used in threads and stacks, constituting shared environment725and/or user programs726. Shared environment725includes operating systems, device drivers, virtual machines, etc., which provide a (common) run time environment for execution of user programs726.

Graphics controller760generates display signals (e.g., in RGB format) to display unit770based on data/instructions received from CPU710. Display unit770contains a display screen to display the images defined by the display signals. Input interface790may correspond to a keyboard and a pointing device (e.g., touch-pad, mouse) and may be used to provide inputs. Network interface780provides connectivity to a network (e.g., using Internet Protocol), and may be used to communicate with other systems (such as those shown inFIG.1) connected to the network104.

Secondary memory730may contain hard drive735, flash memory736, and removable storage drive737. Secondary memory730may store the data software instructions (e.g., for performing the actions noted above with respect to the Figures), which enable digital processing system700to provide several features in accordance with the present disclosure.

Some or all of the data and instructions may be provided on removable storage unit740, and the data and instructions may be read and provided by removable storage drive737to CPU710. Floppy drive, magnetic tape drive, CD-ROM drive, DVD Drive, Flash memory, removable memory chip (PCMCIA Card, EEPROM) are examples of such removable storage drive737.

Removable storage unit740may be implemented using medium and storage format compatible with removable storage drive737such that removable storage drive737can read the data and instructions. Thus, removable storage unit740includes a computer readable (storage) medium having stored therein computer software and/or data. However, the computer (or machine, in general) readable medium can be in other forms.

In this document, the term “computer program product” is used to generally refer to removable storage unit740or hard disk installed in hard drive735. These computer program products are means for providing software to digital processing system700. CPU710may retrieve the software instructions, and execute the instructions to provide various features of the present disclosure described above.

The term “storage media/medium” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or solid-state drives, such as storage memory730. Volatile media includes dynamic memory, such as RAM720. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus (communication path)750. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

In accordance with one or more exemplary embodiments of the present disclosure, a computer program product comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, said program code including instructions to: enable a user to record one or more videos by a video creating and editing module on a computing device; select one or more frames automatically from the one or more user recorded videos by the video creating and editing module; transfer the one or more automatically selected frames from the computing device to a server by the video creating and editing module over a network; receive the one or more automatically selected frames by a video processing module enabled in the server; detect and extract one or more objects from the one or more automatically selected frames by the video processing module; transfer the one or more extracted objects to the computing device by the video processing module over the network; receive the one or more extracted objects by the video creating and editing module and displaying the one or more extracted objects to the user on the computing device; place the one or more extracted objects on a new frame automatically by the video creating and editing module; allow the user to reposition the one or more extracted objects on the new frame by the video creating and editing module; and enable the user to customize background and foreground elements in the new frame to create virtual situations by the video creating and editing module.

According to an exemplary aspect of the present disclosure, enabling a user to record one or more videos by a video creating and editing module on a computing device.

According to an exemplary aspect of the present disclosure, displaying a series of frames to the user from the one or more user recorded videos by the video creating and editing module.

According to an exemplary aspect of the present disclosure, allowing the user to select one or more frames from the series of frames displayed on the computing device by the video creating and editing module.

According to an exemplary aspect of the present disclosure, transferring the one or more user selected frames from the computing device to a server by the video creating and editing module over a network.

According to an exemplary aspect of the present disclosure, receiving the one or more user selected frames by a video processing module enabled in the server.

According to an exemplary aspect of the present disclosure, detecting and extracting one or more objects from the one or more user selected frames by the video processing module.

According to an exemplary aspect of the present disclosure, transferring the one or more extracted objects to the computing device by the video processing module over the network.

According to an exemplary aspect of the present disclosure, receiving the one or more extracted objects by the video creating and editing module and displaying the one or more extracted objects to the user on the computing device.

According to an exemplary aspect of the present disclosure, enabling the user to select a new frame from the one or more user recorded videos by the video creating and editing module.

According to an exemplary aspect of the present disclosure, allowing the user to select and place the one or more extracted objects into the new frame by the video creating and editing module.

According to an exemplary aspect of the present disclosure, enabling the user to customize background and foreground elements in the new frame by the video creating and editing module.

According to an exemplary aspect of the present disclosure, deleting the one or more user selected frames after extracting the one or more objects and replacing one or more user selected frames with the new frame by the video creating and editing module.

Reference throughout this specification to “one embodiment”, “an embodiment”, or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment”, “in an embodiment” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics of the disclosure may be combined in any suitable manner in one or more embodiments. In the above description, numerous specific details are provided such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of embodiments of the disclosure.

Although the present disclosure has been described in terms of certain preferred embodiments and illustrations thereof, other embodiments and modifications to preferred embodiments may be possible that are within the principles and spirit of the invention. The above descriptions and figures are therefore to be regarded as illustrative and not restrictive.

Thus the scope of the present disclosure is defined by the appended claims and includes both combinations and sub-combinations of the various features described hereinabove as well as variations and modifications thereof, which would occur to persons skilled in the art upon reading the foregoing description.