Video tagging by correlating visual features to sound tags

Automatically recommending sound effects based on visual scenes enables sound engineers during video production of computer simulations, such as movies and video games. This recommendation engine may be accomplished by classifying SFX and using a machine learning engine to output a first of the classified SFX for a first computer simulation based on learned correlations between video attributes of the first computer simulation and the classified SFX.

FIELD

The application relates generally to technically inventive, non-routine solutions that are necessarily rooted in computer technology and that produce concrete technical improvements.

BACKGROUND

Machine learning, sometimes referred to as deep learning, can be used for a variety of useful applications related to data understanding, detection, and/or classification. In computer simulation industries such as gaming industries, video and audio are two separate processes. Simulations are first designed and produced without audio, and then audio groups investigate the simulation videos and insert the corresponding sound effects (SFX) from the SFX database, which is time-consuming.

SUMMARY

As understood herein, machine learning may be used to address the technical problem noted above by providing SFX recommendations that are relevant to computer simulation scenes.

Accordingly, an apparatus includes at least one processor and at least one computer storage that is not a transitory signal and that includes instructions executable by the processor to classify sound effects (SFX) to render classified SFX. The instructions are executable to use at least one machine learning engine to output at least a first of the classified SFX for at least a first computer simulation at least in part based on learned correlations between video attributes of the first computer simulation and the classified SFX.

In example embodiments, the instructions may be executable to recommend the first of the classified SFX for the first computer simulation using direct mapping of elements in the first computer simulation to a classification of the first of the classified SFX. In such embodiments, the instructions can be executable to input the first computer simulation without sound to at least a first neural network (NN) trained to learn correlations between visual features in video and SFX tags, and to input to the first NN information from at least a first noisy SFX model comprising ground truth classifications of noisy SFX. The instructions further may be executable to input training data to the first noisy SFX model to train the first noisy SFX model, with the training data including audio clips from one or more computer simulations and synthesized audio clips. The instructions can be further executable to input the training data to plural convolutional NN (CNN) of the first noisy SFX model to render a first output, and then input the first output to a classification mapper that renders a second output comprising predictions of SFX for the first computer simulation.

In some implementations, the first noisy SFX model includes plural gated convolutional neural networks (CNN). At least one bidirectional recurrent neural network (RNN) may be configured to receive output of the plural gated CNN. Also, plural attention-based feed forward neural networks (FNN) can be configured to receive output of the RNN.

In some examples, the first noisy SFX model can include plural gated convolutional neural networks (CNN) at least one of which is configured to receive the training data. At least a first classifier (CLF) network can be configured to receive output of the plural gated CNN, and at least a second CLF network can be configured to receive output of the plural gated CNN. In such embodiments, the first CLF network may be a supervised 32-category network configured for receiving output from the plural gated CNN including data from both the audio clips from one or more computer simulations and the synthesized audio clips. The second CLF network can be a supervised 182-category network configured for receiving output from the plural gated CNN including data from the synthesized audio clips but not from the audio clips from one or more computer simulations.

In another aspect, an apparatus includes at least one processor and at least one computer storage that is not a transitory signal and that includes instructions executable by the processor to train at least a first sound effect (SFX) recommendation engine at least in part by inputting silent video frames and noisy SFX labels to plural residual neural networks (Resnet). The instructions are executable for inputting an output of the Resnet to at least one bi-directional gated recurrent unit to render a vector, and to recommend at least one SFX for at least a first video with no sound at least in part by inputting an output of the Resnet to at least one trained model also configured for receiving as input at least a second video without sound to output at least one SFX tag representing a recommended SFX for the second video.

In another aspect, a method includes classifying first and second sound effects in a first video or a first computer simulation, and based at least in part on the classifying, providing sound effect predictions for a second video or second computer simulation.

The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which:

DETAILED DESCRIPTION

In accordance with present principles, deep learning based domain adaptation methods may be used to recommend SFX for videos and computer simulations such as video games.

The methods described herein may concern multiple objects and multiple actions associated with the multiple objects. For example, an image text-block of many texts may be an “object”, and the type of the image block may be an “action”.

This disclosure also relates generally to computer ecosystems including aspects of consumer electronics (CE) device networks such as but not limited to distributed computer game networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, and artificial neural networks and machine learning applications.

A system herein may include server and client components, connected over a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including AR headsets, VR headsets, game consoles such as Sony PlayStation® and related motherboards, game controllers, portable televisions (e.g. smart TVs, Internet-enabled TVs), portable computers such as laptops and tablet computers, and other mobile devices including smart phones and additional examples discussed below. These client devices may operate with a variety of operating environments. For example, some of the client computers may employ, as examples, Orbis or Linux operating systems, operating systems from Microsoft, or a Unix operating system, or operating systems produced by Apple, Inc. or Google. These operating environments may be used to execute one or more programs/applications, such as a browser made by Microsoft or Google or Mozilla or other browser program that can access websites hosted by the Internet servers discussed below. Also, an operating environment according to present principles may be used to execute one or more computer game programs/applications and other programs/applications that undertake present principles.

A processor may be any conventional general-purpose single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers.

Software modules described by way of the flow charts and user interfaces herein can include various sub-routines, procedures, etc. Without limiting the disclosure, logic stated to be executed by a particular module can be redistributed to other software modules and/or combined together in a single module and/or made available in a shareable library.

Further to what has been alluded to above, logical blocks, modules, and circuits described below can be implemented or performed with a general-purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic device such as an application specific integrated circuit (ASIC), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor can be implemented by a controller or state machine or a combination of computing devices.

The functions and methods described below may be implemented in hardware circuitry or software circuitry. When implemented in software, the functions and methods can be written in an appropriate language such as but not limited to Java, C# or C++, and can be stored on or transmitted through a computer-readable storage medium such as a random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage such as digital versatile disc (DVD), magnetic disk storage or other magnetic storage devices including removable thumb drives, etc. A connection may establish a computer-readable medium. Such connections can include, as examples, hard-wired cables including fiber optics and coaxial wires and digital subscriber line (DSL) and twisted pair wires. Such connections may include wireless communication connections including infrared and radio.

Now specifically referring toFIG. 1, an example system10is shown, which may include one or more of the example devices mentioned above and described further below in accordance with present principles. The first of the example devices included in the system10is a consumer electronics (CE) device such as an audio video device (AVD)12such as but not limited to an Internet-enabled TV with a TV tuner (equivalently, set top box controlling a TV). However, the AVD12alternatively may be an appliance or household item, e.g. computerized Internet enabled refrigerator, washer, or dryer. The AVD12alternatively may also be a computerized Internet enabled (“smart”) telephone, a tablet computer, a notebook computer, an augmented reality (AR) headset, a virtual reality (VR) headset, Internet-enabled or “smart” glasses, another type of wearable computerized device such as a computerized Internet-enabled watch, a computerized Internet-enabled bracelet, a computerized Internet-enabled music player, computerized Internet-enabled head phones, a computerized Internet-enabled implantable device such as an implantable skin device, other computerized Internet-enabled devices, etc. Regardless, it is to be understood that the AVD12is configured to undertake present principles (e.g., communicate with other consumer electronics (CE) devices to undertake present principles, execute the logic described herein, and perform any other functions and/or operations described herein).

Accordingly, to undertake such principles the AVD12can be established by some or all of the components shown inFIG. 1. For example, the AVD12can include one or more displays14that may be implemented by a high definition or ultra-high definition “4K” or higher flat screen and that may be touch-enabled for receiving user input signals via touches on the display. The AVD12may include one or more speakers16for outputting audio in accordance with present principles, and at least one additional input device18such as an audio receiver/microphone for entering audible commands to the AVD12to control the AVD12. The example AVD12may also include one or more network interfaces20for communication over at least one network22such as the Internet, an WAN, an LAN, etc. under control of one or more processors. Thus, the interface20may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as but not limited to a mesh network transceiver. Furthermore, note the network interface20may be, e.g., a wired or wireless modem or router, or other appropriate interface such as, for example, a wireless telephony transceiver, or Wi-Fi transceiver as mentioned above, etc.

It is to be understood that the one or more processors control the AVD12to undertake present principles, including the other elements of the AVD12described herein such as controlling the display14to present images thereon and receiving input therefrom. The one or more processors may include a central processing unit (CPU)24as well as a graphics processing unit (GPU)25on a graphics card25A.

In addition to the foregoing, the AVD12may also include one or more input ports26such as, e.g., a high definition multimedia interface (HDMI) port or a USB port to physically connect (e.g., using a wired connection) to another consumer electronics (CE) device and/or a headphone port to connect headphones to the AVD12for presentation of audio from the AVD12to a user through the headphones. For example, the input port26may be connected via wire or wirelessly to a cable or satellite source26aof audio video content. Thus, the source26amay be, e.g., a separate or integrated set top box, or a satellite receiver. Or, the source26amay be a game console or disk player containing content that might be regarded by a user as a favorite for channel assignation purposes. The source26awhen implemented as a game console may include some or all of the components described below in relation to the CE device44and may implement some or all of the logic described herein.

The AVD12may further include one or more computer memories28such as disk-based or solid-state storage that are not transitory signals, in some cases embodied in the chassis of the AVD as standalone devices or as a personal video recording device (PVR) or video disk player either internal or external to the chassis of the AVD for playing back AV programs or as removable memory media. Also in some embodiments, the AVD12can include a position or location receiver such as but not limited to a cellphone receiver, GPS receiver and/or altimeter30that is configured to, e.g., receive geographic position information from at least one satellite or cellphone tower and provide the information to the processor24and/or determine an altitude at which the AVD12is disposed in conjunction with the processor24. However, it is to be understood that that another suitable position receiver other than a cellphone receiver, GPS receiver and/or altimeter may be used in accordance with present principles to, for example, determine the location of the AVD12in all three dimensions.

Continuing the description of the AVD12, in some embodiments the AVD12may include one or more cameras32that may be, e.g., a thermal imaging camera, a digital camera such as a webcam, an infrared (IR) camera, and/or a camera integrated into the AVD12and controllable by the processor24to generate pictures/images and/or video in accordance with present principles. Also included on the AVD12may be a Bluetooth transceiver34and other Near Field Communication (NFC) element36for communication with other devices using Bluetooth and/or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.

Further still, the AVD12may include one or more auxiliary sensors37(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer, or a magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and/or cadence sensor, a gesture sensor (e.g., for sensing gesture command), etc.) providing input to the processor24. The AVD12may include an over-the-air TV broadcast port38for receiving OTA TV broadcasts providing input to the processor24. In addition to the foregoing, it is noted that the AVD12may also include an infrared (IR) transmitter and/or IR receiver and/or IR transceiver42such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD12.

Still referring toFIG. 1, in addition to the AVD12, the system10may include one or more other consumer electronics (CE) device types. In one example, a first CE device44may be used to send computer game audio and video to the AVD12via commands sent directly to the AVD12and/or through the below-described server while a second CE device46may include similar components as the first CE device44. In the example shown, the second CE device46may be configured as an AR or VR headset worn by a user47as shown. In the example shown, only two CE devices44,46are shown, it being understood that fewer or greater devices may also be used in accordance with present principles.

In the example shown, all three devices12,44,46are assumed to be members of a network such as a secured or encrypted network, an entertainment network or Wi-Fi in, e.g., a home, or at least to be present in proximity to each other in a certain location and able to communicate with each other and with a server as described herein. However, present principles are not limited to a particular location or network unless explicitly claimed otherwise.

The example non-limiting first CE device44may be established by any one of the above-mentioned devices, for example, a smart phone, a digital assistant, a portable wireless laptop computer or notebook computer or game controller (also referred to as “console”), and accordingly may have one or more of the components described below. The second CE device46without limitation may be established by an AR headset, a VR headset, “smart” Internet-enabled glasses, or even a video disk player such as a Blu-ray player, a game console, and the like. Still further, in some embodiments the first CE device44may be a remote control (RC) for, e.g., issuing AV play and pause commands to the AVD12, or it may be a more sophisticated device such as a tablet computer, a game controller communicating via wired or wireless link with a game console implemented by another one of the devices shown inFIG. 1and controlling video game presentation on the AVD12, a personal computer, a wireless telephone, etc.

Accordingly, the first CE device44may include one or more displays50that may be touch-enabled for receiving user input signals via touches on the display50. Additionally or alternatively, the display(s)50may be an at least partially transparent display such as an AR headset display or a “smart” glasses display or “heads up” display, as well as a VR headset display, or other display configured for presenting AR and/or VR images.

The first CE device44may also include one or more speakers52for outputting audio in accordance with present principles, and at least one additional input device54such as, for example, an audio receiver/microphone for entering audible commands to the first CE device44to control the device44. The example first CE device44may further include one or more network interfaces56for communication over the network22under control of one or more CE device processors58. Thus, the interface56may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including mesh network interfaces. It is to be understood that the processor58controls the first CE device44to undertake present principles, including the other elements of the first CE device44described herein such as, e.g., controlling the display50to present images thereon and receiving input therefrom. Furthermore, note that the network interface56may be, for example, a wired or wireless modem or router, or other appropriate interface such as a wireless telephony transceiver, or Wi-Fi transceiver as mentioned above, etc.

Still further, note that in addition to the processor(s)58, the first CE device44may also include a graphics processing unit (GPU)55on a graphics card55A. The graphics processing unit55may be configured for, among other things, presenting AR and/or VR images on the display50.

In addition to the foregoing, the first CE device44may also include one or more input ports60such as, e.g., a HDMI port or a USB port to physically connect (e.g., using a wired connection) to another CE device and/or a headphone port to connect headphones to the first CE device44for presentation of audio from the first CE device44to a user through the headphones. The first CE device44may further include one or more tangible computer readable storage medium62such as disk-based or solid-state storage. Also in some embodiments, the first CE device44can include a position or location receiver such as but not limited to a cellphone and/or GPS receiver and/or altimeter64that is configured to, e.g., receive geographic position information from at least one satellite and/or cell tower, using triangulation, and provide the information to the CE device processor58and/or determine an altitude at which the first CE device44is disposed in conjunction with the CE device processor58. However, it is to be understood that that another suitable position receiver other than a cellphone and/or GPS receiver and/or altimeter may be used in accordance with present principles to, e.g., determine the location of the first CE device44in all three dimensions.

Continuing the description of the first CE device44, in some embodiments the first CE device44may include one or more cameras66that may be, e.g., a thermal imaging camera, an IR camera, a digital camera such as a webcam, and/or another type of camera integrated into the first CE device44and controllable by the CE device processor58to generate pictures/images and/or video in accordance with present principles. Also included on the first CE device44may be a Bluetooth transceiver68and other Near Field Communication (NFC) element70for communication with other devices using Bluetooth and/or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.

Further still, the first CE device44may include one or more auxiliary sensors72(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer, or a magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and/or cadence sensor, a gesture sensor (e.g., for sensing gesture command), etc.) providing input to the CE device processor58. The first CE device44may include still other sensors such as, for example, one or more climate sensors74(e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and/or one or more biometric sensors76providing input to the CE device processor58. In addition to the foregoing, it is noted that in some embodiments the first CE device44may also include an infrared (IR) transmitter and/or IR receiver and/or IR transceiver78such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the first CE device44. The CE device44may communicate with the AVD12through any of the above-described communication modes and related components.

The second CE device46may include some or all of the components shown for the CE device44. Either one or both CE devices may be powered by one or more batteries.

Now in reference to the afore-mentioned at least one server80, it includes at least one server processor82, at least one tangible computer readable storage medium84such as disk-based or solid-state storage. In an implementation, the medium84includes one or more solid state storage drives (SSDs). The server also includes at least one network interface86that allows for communication with the other devices ofFIG. 1over the network22, and indeed may facilitate communication between servers and client devices in accordance with present principles. Note that the network interface86may be, e.g., a wired or wireless modem or router, Wi-Fi transceiver, or other appropriate interface such as a wireless telephony transceiver. The network interface86may be a remote direct memory access (RDMA) interface that directly connects the medium84to a network such as a so-called “fabric” without passing through the server processor82. The network may include an Ethernet network and/or fiber channel network and/or InfiniBand network. Typically, the server80includes multiple processors in multiple computers referred to as “blades” that may be arranged in a physical server “stack”.

Accordingly, in some embodiments the server80may be an Internet server or an entire “server farm”, and may include and perform “cloud” functions such that the devices of the system10may access a “cloud” environment via the server80in example embodiments for, e.g., domain adaptation as disclosed herein. Additionally, or alternatively, the server80may be implemented by one or more game consoles or other computers in the same room as the other devices shown inFIG. 1or nearby.

FIGS. 2 and 3illustrate overall principles. Commencing at block200inFIG. 2, sound effects (SFX) are classified. In an example, this classification may be executed on incoming digitized sound effect signals300to render tags302(graphically shown inFIG. 3) that describe in words the sound effects being classified as set forth elsewhere herein.

Moving to block202inFIG. 2and still cross-referencingFIG. 3, the tags302are registered in a database304. Then, proceeding to block204inFIG. 2, the registered tags may be combined with video without sound306to render video with sound effect sound308. Note that “sound effects” refer to non-verbal audio that is part of computer simulations such as computer games to mimic the sounds of gunfire, fire burning, people running, people yelling exclamations, water, etc. As set forth further below, deep learning/AI techniques are provided herein to assist in sound content creation for computer simulations such as video games.

As used herein, “clean SFX tagging” refers to classifying or tagging clean audio samples (sound effects with a single source of sound) used by game sound designers based on their categories and subcategories, so that they can be registered in a database automatically. This assists the game designers by making search and retrieval during sound mixing more efficient. “Video tagging” refers to recommending sound effects that are relevant to a game scene automatically. This is done to assist game designers by making the sound design process more efficient. Present principles focus on techniques to achieve video tagging.

This disclosure divulges two techniques for video tagging.FIGS. 4-9describe a direct mapping approach in which a deep learning engine is trained to learn a correlation between the visual features of a game video and corresponding SFX (audio) tags302.FIGS. 10-12describe a visual understanding approach in two steps, namely, providing a neural network (NN) to understand the visual content of the game scene and generate visual tags, which includes object tags, action tags, and captions, followed by mapping the visual tags to audio tags using semantic text similarity. Dictionary-based mapping may also be used based on other knowledge bases.

Accordingly and now referring toFIG. 4, in a training phase400video such as computer simulations with SFX sounds402are used to train a NN system to generate tags404for different SFX sources to render SFX tags406. Once the NN system is trained, it may be used in a test phase408to receive video410such as computer simulations without SFX sounds as input to a trained model412described further below to output SFX tags414that are combined with the video410to render video416with SFX sound incorporated therein.

Now referring toFIG. 5, a more detailed explanation of the training phase fromFIG. 4may be seen. Silent video such as computer game video500is input to a trained NN502. A supervised learning approach is used by the NN502for learning a direct mapping between visual features of a video and corresponding sound effects. To train this supervised model, sound annotations for the game audio are required. As understood herein, the process is complicated by the fact that game audio typically contains a mixture of sounds (also referred to as noisy SFX), making it difficult to obtain human annotations504, especially if number of sound categories is large. Hence, a deep learning model506is trained to automatically tag a mixture of sounds (noisy SFX model) to identify the categories of the constituent sounds.

Now referring toFIGS. 6 and 7, in an initial embodiment a noisy SFX model is trained to tag a small number of categories (32 classes) using human annotations. An audio clip represented by the spectrogram600is input to a segmentation mapping module602that includes a series of convolutional NNs (CNNs)604. Segmentation masks606are output by the mapping module602and used for classification mapping608that produces predictions610for tags with corresponding probabilities.FIG. 7relatedly shows a gated convolutional recurrent NN (CRNN)700that receives SFX clips702as input and extracts spectral patterns at each time step, providing output to a bidirectional RNN704such as a bidirectional long short-term memory (LSTM).FIG. 7indicates the types of CNNs used in the network700. The Bi-RNN704is coupled to an attention-based localization module706that includes plural feed forward NNs (FNN) operating as sigmoid and softmax FNN as shown to produce predicted tags708as weighted averages.

Of importance to present principles isFIG. 8, illustrating an advanced technique for noisy SFX tagging. To generate finer-grained SFX tags (e.g., 182-class or 5000-class labels or even more detailed) for better discrimination of different sound effects in a noisy sample, a supervised model is trained using actual SFX data800from computer simulations and synthesized noisy SFX data802generated separately from any simulation solely for purposes of training a gated CNN module804. In other words, present principles as reflected inFIG. 8recognize that to train a supervised model, training data is required that has finer-grained (e.g., 182-class or 5000 class) ground truth tags, whereas only coarser-grained (32-class) human annotated SFX labels for game audio currently is available. Hence,FIG. 8and following figures illustrate a semi-supervised approach that generates fine-grained audio tags from coarse-grained audio tags without additional human annotations. Note that 32-class and 182-class are used as examples of coarse and finer-grained tags.

The synthetic mixtures of sound samples represented at802are created and their categories recorded during mixing. In this synthetic mixture, fine-grained SFX labels (referred to elsewhere herein as Dataset1) are established. Block800, on the other hand, represents the available real game audio with coarse-grained labels (generated by humans) referred to as Dataset2. As shown inFIG. 8, Dataset1of actual simulation or game data and Dataset2of synthesized data that is not from a simulation or game but is created for purposes of supplementing game data are combined to train an end-to-end semi-supervised model804that includes a coarse classifier806and a fine-grained classifier808to generate fine-grained tags810that identify the components of noisy game audio. It is semi-supervised because no true fine-grained game audio labels are present for training, as explained earlier. It is a multi-tasking model because it is capable of generating both coarse-grained audio tags812and fine-grained audio tags810. In other words, fine grain analysis uses more categories than coarse grain analysis.

The training loss function for this model is a sum of the loss for fine-grained tagging and coarse-grained tagging. The goal of the training is to minimize the training loss. The training stops when the model converges. At this point a model is attained that can decompose a noisy audio mixture into its constituent tags.

Accordingly, the above description divulges a technique to identify the constituent sound effect categories of a game audio, whileFIG. 9depicts how to use these tags (generated by human or by the model inFIG. 8) to train a supervised video tagging model. As shown, during training, videos900with sound extracted, along with the noisy SFX tags902generated as described above and/or human-annotated, are input to a training phase module904. With greater specificity, the corresponding audio that is extracted from the video is passed through the noisy SFX model explained above inFIG. 8to generate the SFX tags or labels902, which are input along with the corresponding video segment900to the supervised training phase model904. In this way the video is synchronized with the audio tags before training. In an example non-limiting implementation, the frame rate used may be thirty frames per second (30 fps) and video duration may be one second.

The training phase module904generates video embeddings (numerical vectors) by passing the silent video frames through a deep CNN906(e.g., a Resnet or similar network). For each frame, one embedding (vector) is generated, which serves as the visual feature for the video frame. Other visual features can also be used. Because a video is a sequence of frames, a sequence of video embeddings is produced, which are then input to a recurrent neural network908, in the example shown, a bidirectional gated recurrent unit (GRU) or gated recurrent network that produces tag predictions910.

The output of the training is a neural model912that can receive new simulation video914without sound in a test phase and generate sound tags916corresponding to the silent video914. These tags may be used to retrieve the corresponding sound effects918for combination with the video as shown at920.

FIGS. 10-12illustrate the visual understanding approach alluded to above. In a first step, video1000such as a computer simulation without sound (audio) is used to generate visual tags1002based on visual understanding of, for example, identified objects1004in the video, identified actions1006in the video, and identified scene descriptions1008in the video. Then a semantic text similarity module1010receives the visual tags1002along with SFX tags1012from the database described above to automatically map the visual tags to the specific audio categories in the sound database to generate video1014with sound.

FIG. 11illustrates further. A display1100is shown presenting video with objects1102that are recognized using image recognition techniques to generate corresponding visual tags1104. The visual tags1104may be embedded using word embedding or sentence embedding, which results in a numerical vector. The video tags1104are matched with corresponding audio tags1106. Each audio category or audio file name that identifies an audio sample is embedded using word embedding or sentence embedding, which again results in a numerical vector.FIG. 12similarly shows a video on a display1200with captions1202that can be matched using unsupervised semantic text similarity models1203to audio tags1204.

Using this text similarity approach, each visual tag can be mapped to different granularities of audio tags, ranging from coarse grained (e.g., 32-class) tags that identify a group of audio samples to very fine grained tags that identify an individual sound sample.

The automatically generated audio tags from visual understanding of game scenes can serve two purposes. First, the audio tags can be used to recommend sound effects for game scenes to the game designers. Second, the audio tags can also be used as SFX labels for training the direct mapping video tagging model divulged inFIGS. 4-16as an alternative to the noisy SFX labels derived from audio.

While direct mapping inFIGS. 4-9may provide greater accuracy in tagging than the visual understanding technique shown inFIGS. 10-12, visual understanding renders finer grained tagging using unsupervised text similarity and renders it relatively easy to annotate objects and captions. Direct mapping is particularly advantageous when accurate ground-truth SFX tags are available for tagging or sound source separation is viable. Visual understanding is particularly advantageous when obtaining fine-grained SFX annotations is otherwise difficult, and it mimics the work flow of a sound engineer.

Present principles may be used in deep learning-based methods for image, video and audio data processing, among others. As may be appreciated from the foregoing detailed description, present principles thus improve the adaptation and training of neural networks through the technological solutions described herein.

It will be appreciated that whilst present principals have been described with reference to some example embodiments, these are not intended to be limiting, and that various alternative arrangements may be used to implement the subject matter claimed herein.