Patent Description:
A variety of accessibility features, such as vision compensation, hearing assistance, and neurodiversity tools, for example, can greatly improve the experience of interacting with media content for persons experiencing disabilities. As a specific example, members of the deaf and hearing impaired communities often rely on any of a number of signed languages for communication via hand signals. Although effective in translating the plain meaning of a communication, hand signals alone typically do not fully capture the emphasis or emotional intensity motivating that communication. Accordingly, skilled human sign language translators tend to employ multiple physical modes when communicating information. Those modes may include gestures other than hand signals, postures, and facial expressions, as well as the speed and force with which such expressive movements are executed.

For a human sign language translator, identification of the appropriate emotional intensity and emphasis to include in a signing performance may be largely intuitive, based on cognitive skills honed unconsciously as the understanding of spoken language is learned and refined through childhood and beyond. However, the exclusive reliance on human sign language translation can be expensive, and in some use cases may be inconvenient or even impracticable, while analogous challenges to the provision of vision compensated and neurodiversity sensitive content exist. Consequently, there is a need in the art for an efficient and scalable solution for creating accessibility enhanced content.

<CIT> discloses methods of providing a translation of information including an OCR process applied to an image and/or text that is then converted to the sign language.

<CIT> discloses methods, an apparatus and systems for a sign language recognition.

The dependent claims describe preferred embodiments.

The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals. Moreover, the drawings and illustrations in the present application are generally not to scale, and are not intended to correspond to actual relative dimensions.

The present application discloses systems and methods for creating accessibility enhanced content. It is noted that although the present content enhancement solution is described below in detail by reference to the exemplary use case in which sign language is used to enhance audio-video content having both audio and video components, the present novel and inventive principles may be advantageously applied to video unaccompanied by audio, as well as to audio content unaccompanied by video. In addition, or alternatively, in some implementations, the type of content that is accessibility enhanced according to the present novel and inventive principles may be or include digital representations of persons, fictional characters, locations, objects, and identifiers such as brands and logos, for example, which populate a virtual reality (VR), augmented reality (AR), or mixed reality (MR) environment. Moreover, that content may depict virtual worlds that can be experienced by any number of users synchronously and persistently, while providing continuity of data such as personal identity, user history, entitlements, possessions, payments, and the like. It is noted that the accessibility enhancement solution disclosed by the present application may also be applied to content that is a hybrid of traditional audio-video and fully immersive VR/AR/MR experiences, such as interactive video.

It is further noted that, as defined in the present application, the expression "sign language" refers to any of a number of signed languages relied upon by the deaf community and other hearing impaired persons for communication via hand signals, facial expressions, and in some cases larger body motions or postures. Examples of sign languages within the meaning of the present application include sign languages classified as belonging to the American Sign Language (ASL) cluster, Brazilian sign Language (LIBRAS), the French Sign Language family, Indo-Pakistani Sign Language, Chinese Sign Language, the Japanese Sign Language family, and the British, Australian, and New Zealand Sign Language (BANZSL) family, to name a few.

It is also noted that although the present content enhancement solution is described below in detail by reference to the exemplary use case in which a sign language performance is used to enhance content, the present novel and inventive principles may also be applied to content enhancement through the use of an entire suite of accessibility enhancements. Examples of such accessibility enhancements include assisted audio, forced narratives, subtitles, captioning, and the provision of haptic effects, to name a few. Moreover, in some implementations, the systems and methods disclosed by the present application may be substantially or fully automated.

As used in the present application, the terms "automation," "automated," and "automating" refer to systems and processes that do not require the participation of a human analyst or editor. Although, in some implementations, a human system administrator may sample or otherwise review the accessibility enhanced content distributed by the automated systems and according to the automated methods described herein, that human involvement is optional. Thus, the methods described in the present application may be performed under the control of hardware processing components of the disclosed automated systems.

It is also noted that, as defined in the present application, the expression "machine learning model" may refer to a mathematical model for making future predictions based on patterns learned from samples of data or "training data. " For example, machine learning models may be trained to perform image processing, natural language processing (NLP), and other inferential processing tasks. Various learning algorithms can be used to map correlations between input data and output data. These correlations form the mathematical model that can be used to make future predictions on new input data. Such a predictive model may include one or more logistic regression models, Bayesian models, or artificial neural networks (NNs). A "deep neural network," in the context of deep learning, may refer to an NN that utilizes multiple hidden layers between input and output layers, which may allow for learning based on features not explicitly defined in raw data. As used in the present application, a feature identified as an NN refers to a deep neural network.

<FIG> shows exemplary system <NUM> for creating accessibility enhanced content, according to one implementation. As shown in <FIG>, system <NUM> includes computing platform <NUM> having processing hardware <NUM> and system memory <NUM> implemented as a computer-readable non-transitory storage medium. According to the present exemplary implementation, system memory <NUM> stores software code <NUM> that may include one or more machine learning models, as well as performer database <NUM>, word string database <NUM>, and video tokens database <NUM>.

As further shown in <FIG>, system <NUM> is implemented within a use environment including content broadcast source <NUM> providing primary content <NUM> to system <NUM> and receiving accessibility enhanced content <NUM> corresponding to primary content <NUM> from system <NUM>. With respect to the feature "performer database," as defined for the purposes of the present application the term "performer" refers to a digital representation of an actor, or a virtual character such as an animated model or cartoon for example, that delivers or "performs" an accessibility enhancement, such as narration, voice-over, or a sign language interpretation of primary content <NUM>.

In addition, as defined for the purposes of the present application, the feature "word string" may refer to a single word or a phrase including a sequence of two or more words. Moreover, in some implementations, a word string entry in word string database <NUM> may include, in addition to a particular word string, one or more of the probability of that word string corresponding to a particular emotive state, physical gestures or facial expressions corresponding to the word string, or haptic effects associated with the word string.

Regarding the feature "video tokens," it is noted that as defined in the present application, a "video token" refers to a snippet of video content including a predetermined accessibility enhancement. In the exemplary use case of content enhanced using by a performance of a sign language translation (hereinafter "sign language performance"), for example, single word signs, certain commonly used sequences of signs, or commonly recognized shorthand representations of lengthy sequences of signs may be pre-produced as video tokens to be played back when primary content <NUM> reaches a location corresponding respectively to each video token.

As depicted in <FIG>, in some use cases, content broadcast source <NUM> may find it advantageous or desirable to make primary content <NUM> available via an alternative distribution mode, such as communication network <NUM>, which may take the form of a packet-switched network, for example, such as the Internet. For instance, system <NUM> may be utilized by content broadcast source <NUM> to distribute accessibility enhanced content <NUM> including primary content <NUM> as part of a content stream, which may be an Internet Protocol (IP) content stream provided by a streaming service, or a video-on-demand (VOD) service.

The use environment of system <NUM> also includes user systems 140a, 140b, and 140c (hereinafter "user systems 140a-140c") receiving accessibility enhanced content <NUM> from system <NUM> via communication network <NUM>. With respect to user systems 140a-140c, it is noted that although <FIG> depicts three user systems, that representation is merely by way of example. In other implementations, user systems 140a-140c may include as few as one user system, or more than three user systems.

Also shown in <FIG> are network communication links <NUM> of communication network <NUM> interactively connecting system <NUM> with user systems 140a-140c, as well as displays 148a, 148b, and 148c (hereinafter "displays 148a-148c") of respective user systems 140a-140c. As discussed in greater detail below, accessibility enhanced content <NUM> includes primary content <NUM> as well as an accessibility track synchronized to primary content <NUM>. In some implementations, for example, such an accessibility track may include imagery depicting a performance of a sign language translation of primary content <NUM> for rendering on one or more of displays 148a-148c.

Although the present application refers to software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM> as being stored in system memory <NUM> for conceptual clarity, more generally, system memory <NUM> may take the form of any computer-readable non-transitory storage medium. The expression "computer-readable non-transitory storage medium," as used in the present application, refers to any medium, excluding a carrier wave or other transitory signal that provides instructions to processing hardware <NUM> of computing platform <NUM> or to respective processing hardware of user systems 140a-140c. Thus, a computer-readable non-transitory storage medium may correspond to various types of media, such as volatile media and non-volatile media, for example. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical discs such as DVDs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.

Moreover, although <FIG> depicts to software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM> as being co-located in system memory <NUM>, that representation is also provided merely as an aid to conceptual clarity. More generally, system <NUM> may include one or more computing platforms <NUM>, such as computer servers for example, which may be co-located, or may form an interactively linked but distributed system, such as a cloud-based system, for instance. As a result, processing hardware <NUM> and system memory <NUM> may correspond to distributed processor and memory resources within system <NUM>. Consequently, in some implementations, one or more of software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM> may be stored remotely from one another on the distributed memory resources of system <NUM>.

Processing hardware <NUM> may include multiple hardware processing units, such as one or more central processing units, one or more graphics processing units, and one or more tensor processing units, one or more field-programmable gate arrays (FPGAs), custom hardware for machine-learning training or inferencing, and an application programming interface (API) server, for example. By way of definition, as used in the present application, the terms "central processing unit" (CPU), "graphics processing unit" (GPU), and "tensor processing unit" (TPU) have their customary meaning in the art. That is to say, a CPU includes an Arithmetic Logic Unit (ALU) for carrying out the arithmetic and logical operations of computing platform <NUM>, as well as a Control Unit (CU) for retrieving programs, such as software code <NUM>, from system memory <NUM>, while a GPU may be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. A TPU is an application-specific integrated circuit (ASIC) configured specifically for artificial intelligence (AI) processes such as machine learning.

In some implementations, computing platform <NUM> may correspond to one or more web servers accessible over a packet-switched network such as the Internet, for example. Alternatively, computing platform <NUM> may correspond to one or more computer servers supporting a wide area network (WAN), a local area network (LAN), or included in another type of private or limited distribution network. In addition, or alternatively, in some implementations, system <NUM> may utilize a local area broadcast method, such as User Datagram Protocol (UDP) or Bluetooth, for instance. Furthermore, in some implementations, system <NUM> may be implemented virtually, such as in a data center. For example, in some implementations, system <NUM> may be implemented in software, or as virtual machines.

It is further noted that, although user systems 140a-140c are shown variously as desktop computer 140a, smartphone 140b, and smart television (smart TV) 140c, in <FIG>, those representations are provided merely by way of example. In other implementations, user systems 140a-140c may take the form of any suitable mobile or stationary computing devices or systems that implement data processing capabilities sufficient to provide a user interface, support connections to communication network <NUM>, and implement the functionality ascribed to user systems 140a-140c herein. That is to say, in other implementations, one or more of user systems 140a-140c may take the form of a laptop computer, tablet computer, digital media player, game console, or a wearable communication device such as a smartwatch, AR viewer, or VR headset, to name a few examples. It is also noted that displays 148a-148c may take the form of liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, quantum dot (QD) displays, or any other suitable display screens that perform a physical transformation of signals to light.

In some implementations, content broadcast source <NUM> may be a media entity providing primary content <NUM>. Primary content <NUM> may include content from a linear TV program stream, for example, that includes a high-definition (HD) or ultra-HD (UHD) baseband video signal with embedded audio, captions, time code, and other ancillary metadata, such as ratings and/or parental guidelines. In some implementations, primary content <NUM> may also include multiple audio tracks, and may utilize secondary audio programming (SAP) and/or Descriptive Video Service (DVS), for example. Alternatively, in some implementations, primary content <NUM> may be video game content. As yet another alternative, and as noted above, in some implementations primary content <NUM> may be or include digital representations of persons, fictional characters, locations, objects, and identifiers such as brands and logos, for example, which populate a VR, AR, or MR environment. Moreover, primary content <NUM> may depict virtual worlds that can be experienced by any number of users synchronously and persistently, while providing continuity of data such as personal identity, user history, entitlements, possessions, payments, and the like. As also noted above, primary content <NUM> may be or include content that is a hybrid of traditional audio-video and fully immersive VR/AR/MR experiences, such as interactive video.

In some implementations, primary content <NUM> may be the same source video that is broadcast to a traditional TV audience. Thus, content broadcast source <NUM> may take the form of a conventional cable and/or satellite TV network, for example. As noted above, content broadcast source <NUM> may find it advantageous or desirable to make primary content <NUM> available via an alternative distribution mode, such as communication network <NUM>, which may take the form of a packet-switched network, for example, such as the Internet, as also noted above. Alternatively, or in addition, although not depicted in <FIG>, in some use cases accessibility enhanced content <NUM> may be distributed on a physical medium, such as a DVD, Blu-ray Disc®, or FLASH drive, for example.

<FIG> shows another exemplary system, i.e., user system <NUM>, for use in creating accessibility enhanced content, according to one implementation. As shown in <FIG>, user system <NUM> includes computing platform <NUM> having transceiver <NUM>, processing hardware <NUM>, user system memory <NUM> implemented as a computer-readable non-transitory storage medium, and display <NUM>. As further shown in <FIG>, user system memory <NUM> stores software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM>. With respect to display <NUM>, it is noted that, in various implementations, display <NUM> may be physically integrated with user system <NUM> or may be communicatively coupled to but physically separate from user system <NUM>. For example, where user system <NUM> is implemented as a smart TV, smartphone, laptop computer, tablet computer, AR viewer, or VR headset, display <NUM> will typically be integrated with user system <NUM>. By contrast, where user system <NUM> is implemented as a desktop computer, display <NUM> may take the form of a monitor separate from computing platform <NUM> in the form of a computer tower.

As also shown in <FIG>, user system <NUM> is utilized in use environment <NUM> including content broadcast source <NUM> providing primary content <NUM> to content distribution network <NUM>, which in turn distributes primary content <NUM> to user system <NUM> via communication network <NUM> and network communication links <NUM>. According to the implementation shown in <FIG>, software code <NUM> stored in user system memory <NUM> of user system <NUM> is configured to receive primary content <NUM> and to output accessibility enhanced content <NUM> including primary content <NUM> for rendering on display <NUM>.

Content broadcast source <NUM>, primary content <NUM>, accessibility enhanced content <NUM>, communication network <NUM>, and network communication links <NUM> correspond respectively in general to content broadcast source <NUM>, primary content <NUM>, accessibility enhanced content <NUM>, communication network <NUM>, and network communication links <NUM>, in <FIG>. In other words, content broadcast source <NUM>, primary content <NUM>, accessibility enhanced content <NUM>, communication network <NUM>, and network communication links <NUM> may share any of the characteristics attributed to respective content broadcast source <NUM>, primary content <NUM>, accessibility enhanced content <NUM>, communication network <NUM>, and network communication links <NUM> by the present disclosure, and vice versa.

User system <NUM> and display <NUM> correspond respectively in general to any or all of user systems 140a-140c and respective displays 148a-148c in <FIG>. Thus, user systems 140a-140c and displays 148a-148c may share any of the characteristics attributed to respective user system <NUM> and display <NUM> by the present disclosure, and vice versa. That is to say, like displays 148a-148c, display <NUM> may take the form of an LCD, LED display, OLED display, or QD display, for example. Moreover, although not shown in <FIG>, each of user systems 140a-140c may include features corresponding respectively to computing platform <NUM>, transceiver <NUM>, processing hardware <NUM>, and user system memory <NUM> storing software code <NUM>.

Transceiver <NUM> may be implemented as a wireless communication unit configured for use with one or more of a variety of wireless communication protocols. For example, transceiver <NUM> may be implemented as a fourth generation (<NUM>) wireless transceiver, or as a <NUM> wireless transceiver. In addition, or alternatively, transceiver <NUM> may be configured for communications using one or more of WiFi, Bluetooth, Bluetooth LE, ZigBee, and <NUM> wireless communications methods.

User system processing hardware <NUM> may include multiple hardware processing units, such as one or more CPUs, one or more GPUs, one or more TPUs, and one or more FPGAs, for example, as those features are defined above.

Software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM> correspond respectively in general to software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM>, in <FIG>. Thus, software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM>, may share any of the characteristics attributed to respective software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM> by the present disclosure, and vice versa. In other words, like software code <NUM>, software code may include one or more machine learning models. Moreover, in implementations in which client processing hardware <NUM> executes software code <NUM> stored locally in user system memory <NUM>, user system <NUM> may perform any of the actions attributed to system <NUM> by the present disclosure. Thus, in some implementations, software code <NUM> executed by processing hardware <NUM> of user system <NUM> may receive primary content <NUM> and may output accessibility enhanced content <NUM> including primary content <NUM> and an accessibility track synchronized to primary content <NUM>.

<FIG> shows an exemplary implementation in which accessibility enhanced content <NUM> is provided to one or more viewers via user system <NUM>. As shown in <FIG>, accessibility enhanced content <NUM> includes primary content <NUM> and sign language translation <NUM> of primary content <NUM>, shown as an overlay of primary content <NUM> on display <NUM>. User system <NUM>, display <NUM>, primary content <NUM>, and accessibility enhanced content <NUM> correspond respectively in general to user system(s) 140a-140c/<NUM>, display(s) 148a-148c/<NUM>, primary content <NUM>/<NUM>, and accessibility enhanced content <NUM>/<NUM> in <FIG> and <FIG>. As a result, user system <NUM>, display <NUM>, primary content <NUM>, and accessibility enhanced content <NUM> may share any of the characteristics attributed to respective user system(s) 140a-140c/<NUM>, display(s) 148a-148c/<NUM>, primary content <NUM>/<NUM>, and accessibility enhanced content <NUM>/<NUM> by the present disclosure, and vice versa. That is to say, like display(s) 148a-148c/<NUM>, display <NUM> may take the form of an LCD, LED display, OLED display, QD display, or any other suitable display screen that performs a physical transformation of signals to light. In addition, although not shown in <FIG>, user system <NUM> may include features corresponding respectively to computing platform <NUM>, processing hardware <NUM>, and system memory storing software code <NUM>, performer database <NUM>, word string database <NUM>, and video tokens database <NUM>, in <FIG>.

It is noted that although sign language translation <NUM> of primary content <NUM>, is shown as an overlay of primary content <NUM>, in <FIG>, that representation is merely exemplary. In other implementations, the display dimensions of primary content <NUM>/<NUM>/<NUM> may be reduced so as to allow sign language translation <NUM> of primary content <NUM>/<NUM> to be rendered next to primary content <NUM>/<NUM>/<NUM>, e.g., above, below, or laterally adjacent to primary content <NUM>/<NUM>/<NUM>. Alternatively, in some implementations, sign language translation <NUM> of primary content <NUM>/<NUM>/<NUM> may be projected or otherwise displayed on a surface other than display 148a-148c/<NUM>/<NUM>, such as a projection screen or wall behind or next to user system 140a-140c/<NUM>/<NUM>, for example.

Sign language translation <NUM> of primary content <NUM>/<NUM>/<NUM> may be performed by a performer in the form of a digital representation of an actor a computer generated digital character (hereinafter "animated model"), such as an animated cartoon for example. For instance, software code <NUM>/<NUM> may be configured to programmatically interpret one or more of visual images, audio, a script, captions, subtitles, or metadata of primary content <NUM>/<NUM>/<NUM> into sign language hand signals, as well as other gestures, postures, and facial expressions communicating a message conveyed by content <NUM>/<NUM>/<NUM>, and to perform that interpretation using the performer. It is noted that background music with lyrics can be distinguished from lyrics being sung by a character using facial recognition, object recognition, activity recognition, or any combination of those technologies performed by software code <NUM>/<NUM>, for example using one or more machine learning model-based analyzers included in software code <NUM>/<NUM>. It is further noted that software code <NUM>/<NUM> may be configured to predict appropriate facial expressions and postures for execution by the performer during performance of sign language translation <NUM>, as well as to predict the speed and forcefulness or emphasis with which the performer executes the performance of sign language translation <NUM>.

Referring to <FIG> and <FIG> in combination, in some implementations, processing hardware <NUM> of computing platform <NUM> may execute software code <NUM> to synchronize sign language translation <NUM> with a timecode of primary content <NUM>/<NUM> when producing accessibility enhanced content <NUM>/<NUM>, and to record accessibility enhanced content <NUM>/<NUM>, or to broadcast or stream accessibility enhanced content <NUM>/<NUM> to user system 140a-140c/<NUM>. In some of those implementations, the performance of sign language translation <NUM> by the performer may be pre-rendered by system <NUM> and broadcasted or streamed to user system 148a-148c/<NUM>. However, in other implementations in which accessibility enhanced content <NUM>/<NUM> including primary content <NUM>/<NUM> and sign language translation <NUM> are broadcasted or streamed to user system 140a-140c/<NUM>, processing hardware <NUM> may execute software code <NUM> to generate sign language translation <NUM> dynamically during the recording, broadcasting, or streaming of primary content <NUM>/<NUM>.

Further referring to <FIG>, in yet other implementations in which primary content <NUM>/<NUM> is broadcasted or streamed to user system <NUM>/<NUM>, processing hardware <NUM> of user system <NUM>/<NUM> may execute software code <NUM> to generate sign language translation <NUM> locally on user system <NUM>/<NUM>, and to do so dynamically during play back of primary content <NUM>/<NUM>. Processing hardware <NUM> of user system <NUM>/<NUM> may further execute software code <NUM> to render the performance of sign language translation <NUM> on display <NUM>/<NUM> contemporaneously with rendering primary content <NUM>/<NUM>.

In some implementations, the pre-rendered performance of sign language translation <NUM> by a performer, or facial points and other digital character landmarks for performing sign language translation <NUM> dynamically using the performer may be transmitted to user system(s) 140a-140c/<NUM>/<NUM> using a separate communication channel than that used to send and receive primary content <NUM>/<NUM>/<NUM>. In one such implementation, the data for use in performing sign language translation <NUM> may be generated by software code <NUM> on system <NUM>, and may be transmitted to user system(s) 140a-140c/<NUM>/<NUM>. In other implementations, the data for use in performing sign language translation <NUM> may be generated locally on user system <NUM>/<NUM> by software code <NUM>, executed by processing hardware <NUM>.

According to some implementations, multiple channels can be used to transmit sign language performance <NUM>. For example, in some use cases primary content may include dialogue including multiple interactive conversations among two or more participant. In some such use cases, sign language performance <NUM> may include multiple performers, each corresponding respectively to one of the multiple participants. Moreover, in some use cases, the performance by each individual performer may be transmitted to user system(s) 140a-140c/<NUM>/<NUM> on separate communication channels.

In some implementations, it may be advantageous or desirable to enable a user of user system(s) 140a-140c/<NUM>/<NUM> to affirmatively select a particular performer to perform sign language translation <NUM> from a predetermined cast of selectable performers. In those implementations, a child user could select an age appropriate performer different from a performer selected by an adult user. Alternatively, or in addition, the cast of selectable performers may vary depending on the subject matter of primary content <NUM>/<NUM>/<NUM>. For instance, where primary content <NUM>/<NUM>/<NUM> portrays a sporting event, the selectable or default performer for performing sign language translation <NUM> may depict athletes, while actors or fictional characters may be depicted by sign language translation <NUM> when primary content <NUM>/<NUM>/<NUM> is a movie or episodic TV content.

In some implementations, sign language performance <NUM> may include a full-length video of a performer signing the audio of primary content <NUM>/<NUM>/<NUM>, or can include a set of short video tokens each depicting single word signs, certain commonly used sequences of signs, or commonly recognized shorthand representations of lengthy sequences of signs, as noted above. Primary content <NUM>/<NUM>/<NUM> may have a dedicated layer for delivering sign language performance <NUM>. Where sign language performance <NUM> includes the full-length video, sign language performance <NUM> may be streamed contemporaneously with streaming of primary content <NUM>/<NUM>/<NUM>, and may be synchronized to a subtitle track of primary content <NUM>/<NUM>/<NUM>, for example. In some implementations, such a dedicated sign language layer can be toggled on/off. Where sign language performance <NUM> includes a set of video tokens, those video tokens may be delivered to and stored on user system(s) 140a-140c/<NUM>/<NUM>, and a video token can be played back when the subtitle track reaches a corresponding word or phrase, for example. In some implementations, sign language performance <NUM> may be displayed as a picture-in-picture (PiP) overlay on primary content <NUM>/<NUM>/<NUM> that can be repositioned or toggled on/off based on a user selection. The PiP overlay of sign language performance <NUM> can employ alpha masking (green-screening) to show only the performer of sign language performance <NUM>, or the performer having an outline added for contrast.

In some implementations, sign language performance <NUM> may be derived from audio of primary content <NUM>/<NUM>/<NUM> using natural language processing (NLP). Sign language performance <NUM> may also be derived from subtitles or closed captioning of primary content <NUM>/<NUM>/<NUM> using text recognition. In some implementations, sign language performance <NUM> may be computer generated and displayed utilizing an animated model, as noted above. Instructions for rendering the animated model and its animations may be delivered to user system(s) 140a-140c/<NUM>/<NUM>, and the animated model may be rendered on user system(s) 140a-140c/<NUM>/<NUM>. Alternatively, the animated model and its animations may be partially or fully pre-rendered and delivered to user system(s) 140a-140c/<NUM>/<NUM>. Bandwidth and caching capabilities can be checked before delivering pre-rendered models or animations. The animated model and its animations may be display as a PiP overlay.

Video tokens database <NUM> of system <NUM>, or video tokens database <NUM> of user system(s) 140a-140c/<NUM>/<NUM> may include animated performances of commonly used signs with multiple performances available for each sign or sequence of signs depending on the emotion of the performance. The choice of which performance is selected for a given word or phrase could then be determined by another data set that is delivered to user system(s) 140a-140c/<NUM>/<NUM>. The performances may be captured for a standard humanoid rig or multiple humanoid rigs with varying proportions, and then dynamically applied to any animated model with the same proportions, as a way to allow a programmer or user to select which animated model will perform the sign.

In implementations in which primary content <NUM>/<NUM>/<NUM> includes location information, such as from sports cameras or other two-dimensional (2D) or three-dimensional (3D) cameras, a performer for performing sign language performance <NUM> may be inserted into primary content <NUM>/<NUM>/<NUM>, rather than simply overlaid on primary content <NUM>/<NUM>/<NUM>. For example, the performer could be inserted into primary content <NUM>/<NUM>/<NUM> at various depths, or behind various objects. The performer inserted into primary content <NUM>/<NUM>/<NUM> could appear to maintain its respective orientation, e.g., facing a football field, as the camera moves in a given scene, or could change its orientation during the scene to always face the camera. Where primary content <NUM>/<NUM>/<NUM> includes color awareness, such as DOLBY VISION®, the performer may dynamically adapt to colors of primary content <NUM>/<NUM>/<NUM>. For example, grading can be applied to the performer in order for the performer to blend in with primary content <NUM>/<NUM>/<NUM>, or grading can be removed from the performer in order to create contrast with primary content <NUM>/<NUM>/<NUM>. The performer may continually adapt to different colors as primary content <NUM>/<NUM>/<NUM> plays. As another example, where a sign language performance <NUM> PiP overlay is located in the bottom right of display 148a-148c/<NUM>/<NUM>, as action begins to occur in the bottom right, the PiP overlay can be relocated to the bottom left.

In some implementations, a first data set may be utilized to control the performer to perform signing, e.g., with its hands and arms. The first data set can be derived from primary content <NUM>/<NUM>/<NUM>, e.g., from text recognition of the subtitles, closed captioning, NLP of the audio, or any combination thereof. A second data set (hereinafter "emotive data set") can be utilized to control the performer to perform emotions, e.g., facial expressions and other gestures. Such an emotive data set may be a collection of metadata tags that adhere to a predefined taxonomy and are attached to specific timestamps or timecode intervals, for example. Alternatively, in some implementations, the metadata tag definitions themselves may be delivered and loaded when primary content <NUM>/<NUM>/<NUM> is played back, thereby advantageously allowing the taxonomy to be refined or improved over time.

The emotive data set can be derived from facial scanning or similar technologies. The emotive data set may also be derived from expression metadata tags in an expressions track of primary content <NUM>/<NUM>/<NUM>. Expression metadata tags may be manually added by users. Over time, machine learning can be utilized to automate generation of expression metadata tags. The emotive data set can also be derived from audio recognition of primary content <NUM>/<NUM>/<NUM>. For example, if audio data detects an emotional song, the performer may perform a more emotional facial expression. As noted above, system <NUM> may include video tokens database <NUM>, or user system(s) 140a-140c/<NUM>/<NUM> may include video tokens database <NUM>, of performances of commonly used signs or sequences of signs, with multiple performances available for each sign or sequence of signs depending on the emotion of the performance. The choice of which performance is selected for a given word could then be determined based on the emotive data set.

In use cases in which a performer is experiencing multiple emotions concurrently, several alternatives for expressing that complex emotional state may be employed. In some use cases, a video token or performance identified by the same metadata tags as the desired emotional state may already exist. Alternatively, existing performances that collectively include the metadata tags of the desired emotional state could be blended together and applied to a performer. As another alternative, the emotive data set may include weights for the individual expression tags at each timecode. In this use case, a video token or performance could be chosen that contains the emotion tag corresponding to the expression tag with the highest weight. As yet another alternative, there could be predefined business logic for which emotion tags are most important, such as by assigning a predetermined weight to each. In this use case, the video token or performance could be chosen that contains the emotion tag with the highest weight. For example, perhaps "anger" has a higher weight than "tiredness," such that a performer that is concurrently angry and tired executes a performance that conveys anger rather than tiredness.

In some implementations, primary content <NUM>/<NUM>/<NUM> stream may include dedicated channels for senses other than hearing and sight, such as a dedicated haptics effects channel. Users may receive haptic effects based on what occurs in primary content <NUM>/<NUM>/<NUM>. For example, an explosion sound can trigger a shaking haptic effect. Technologies being developed may allow for digital expressions of the sense of taste, and primary content <NUM>/<NUM>/<NUM> stream can include a dedicated taste channel.

The functionality of system <NUM>, user system(s) 140a-140c/<NUM>/<NUM>, and software code <NUM>/<NUM> shown variously in <FIG>, <FIG>, and <FIG> will be further described by reference to <FIG> shows flowchart <NUM> presenting an exemplary method for creating accessibility enhanced content, according to one implementation. With respect to the method outlined in <FIG>, it is noted that certain details and features have been left out of flowchart <NUM> in order not to obscure the discussion of the inventive features in the present application.

Referring to <FIG> in combination with <FIG> and <FIG> flowchart <NUM> begins with receiving primary content <NUM>/<NUM> (action <NUM>). As noted above, primary content <NUM>/<NUM> may include content in the form of video games, music videos, animation, movies, or episodic TV content that includes episodes of TV shows that are broadcasted, streamed, or otherwise available for download or purchase on the Internet or via a user application. Alternatively, or in addition, primary content <NUM>/<NUM> may be or include digital representations of persons, fictional characters, locations, objects, and identifiers such as brands and logos, for example, which populate a VR, AR, or MR environment. Moreover, primary content <NUM>/<NUM> may depict virtual worlds that can be experienced by any number of users synchronously and persistently, while providing continuity of data such as personal identity, user history, entitlements, possessions, payments, and the like. As also noted above, primary content <NUM>/<NUM> may be or include content that is a hybrid of traditional audio-video and fully immersive VR/AR/MR experiences, such as interactive video.

As shown in <FIG>, in some implementations, primary content <NUM> may be received by system <NUM> from broadcast source <NUM>. In those implementations, primary content <NUM> may be received by software code <NUM>, executed by processing hardware <NUM> of computing platform <NUM>. As shown in <FIG>, in other implementations, primary content <NUM> may be received by user system <NUM> from content distribution network <NUM> via communication network <NUM> and network communication links <NUM>. Referring to <FIG>, in those implementations, primary content <NUM> may be received by software code <NUM>, executed by processing hardware <NUM> of user system computing platform <NUM>.

Flowchart <NUM> further includes executing at least one of a visual analysis or an audio analysis of primary content <NUM>/<NUM> (action <NUM>). For example, processing hardware <NUM>/<NUM> may execute software code <NUM>/<NUM> to utilize a visual analyzer included as a feature of software code <NUM>/<NUM>, an audio analyzer included as a feature of software code <NUM>/<NUM>, or such a visual analyzer and audio analyzer, to perform the analysis of primary content <NUM>/<NUM> in action <NUM>.

In various implementations, a visual analyzer included as a feature of software code <NUM>/<NUM> may be configured to apply computer vision or other AI techniques to primary content <NUM>/<NUM>, or may be implemented as an NN or other type of machine learning model. Such a visual analyzer may be configured or trained to recognize which characters are speaking, as well as the intensity of their delivery. In particular, such a visual analyzer may be configured or trained to identify humans, characters, or other talking animated objects, and identify emotions or intensity of messaging. In various use cases, different implementations of such a visual analyzer may be used for different types of content (i.e., a specific configuration or training for specific content). For example, for a news broadcast, the visual analyzer may be configured or trained to identify specific TV anchors and their characteristics, or salient regions of frames within video content for the visual analyzer to focus on may be specified, such as regions in which the TV anchor usually is seated.

An audio analyzer included as a feature of software code <NUM>/<NUM> may also be implemented as an NN or other machine learning model. As noted above, in some implementations, a visual analyzer and an audio analyzer may be used in combination to analyze primary content <NUM>/<NUM>. For instance, in analyzing a football game or other sporting event, the audio analyzer can be configured or trained to listen to the audio track of the event, and its analysis may be verified using the visual analyzer or the visual analyzer may interpret the video of the event, and its analysis may be verified using the audio analyzer. It is noted that primary content <NUM>/<NUM> will typically include multiple video frames and multiple audio frames. In some of those use cases, processing hardware <NUM> may execute software code <NUM>, or processing hardware <NUM> may execute software code <NUM> to perform the visual analysis of primary content <NUM>/<NUM>, the audio analysis of primary content <NUM>/<NUM>, or both the visual analysis and the audio analysis, on a frame-by-frame basis.

In some use cases, primary content <NUM>/<NUM> may include text, such as subtitles or other captioning for example. In use cases in which primary content <NUM>/<NUM> includes text, processing hardware <NUM>/<NUM> may further execute software code <NUM>/<NUM> to utilize a text analyzer included as a feature of software code <NUM>/<NUM> to analyze primary content <NUM>/<NUM>. Thus, in use cases in which primary content <NUM>/<NUM> includes text, action <NUM> may further include analyzing that text.

It is further noted that, in some use cases, primary content <NUM>/<NUM> may include metadata. In use cases in which primary content <NUM>/<NUM> includes metadata, processing hardware <NUM>/<NUM> may execute software code <NUM>/<NUM> to utilize a metadata parser included as a feature of software code <NUM>/<NUM> to extract metadata from primary content <NUM>/<NUM>. Thus, in use cases in which primary content <NUM>/<NUM> includes metadata, action <NUM> may further include extracting and analyzing that metadata.

Referring to <FIG> in combination with <FIG>, flowchart <NUM> further includes generating, based on executing the at least one of the visual analysis or the audio analysis in action <NUM>, an accessibility track synchronized to primary content <NUM>/<NUM>/<NUM> (action <NUM>). Such an accessibility track includes a sign language performance <NUM>, optionally a video token or video tokens configured to be played back when primary content <NUM>/<NUM>/<NUM> reaches a location, such as a timestamp or timecode interval, for example, corresponding to each of the video token of tokens, or one or more haptic effects configured to be actuated when primary content <NUM>/<NUM>/<NUM> reaches a location corresponding to each of the one or more haptic effects. It is noted that, in some implementations, one or more video tokens may be played back, or one or more haptic effects may be actuated, dynamically, in response to a particular word or words being spoken or in response to the presence of a particular sound in primary content <NUM>/<NUM>/<NUM>.

It is noted that, in some implementations, action <NUM> may include first generating the accessibility track and subsequently synchronizing the accessibility track to primary content <NUM>/<NUM>/<NUM>, while in other implementations the generation of the accessibility track and its synchronization to primary content <NUM>/<NUM>/<NUM> may be performed contemporaneously. It is further noted that, in various implementations, the accessibility track generated in action <NUM> may be synchronized with the timecode of primary content <NUM>/<NUM>/<NUM>, a subtitle track of primary content <NUM>/<NUM>/<NUM>, an audio track of primary content <NUM>/<NUM>/<NUM>, or to individual frames or sequences of frames of primary content <NUM>/<NUM>/<NUM>. Generation of the accessibility track, or the generation and subsequent synchronization of the accessibility track to primary content <NUM>/<NUM>/<NUM>, in action <NUM>, may be performed by software code <NUM> executed by processing hardware <NUM> of system <NUM>, or by software code <NUM> executed by processing hardware <NUM> of user system <NUM>.

Flowchart <NUM> further includes supplementing primary content <NUM>/<NUM>/<NUM> with the accessibility track generated in action <NUM> to provide accessibility enhanced content <NUM>/<NUM>/<NUM> (action <NUM>). Action <NUM> may be performed by software code <NUM> executed by processing hardware <NUM> of system <NUM>, or by software code <NUM> executed by processing hardware <NUM> of user system <NUM>/<NUM>.

As discussed above by reference to <FIG> and <FIG>, in some implementations, processing hardware <NUM> of system <NUM> may executed software code <NUM> to broadcast or stream accessibility enhanced content <NUM>/<NUM> including synchronized sign language performance <NUM> to user system(s) 140a-140c/<NUM>. In some of those implementations, the performance of sign language translation <NUM> may be pre-rendered by system <NUM> and broadcasted or streamed to user system(s) 140a-140c/<NUM>. However, in other implementations in which primary content <NUM>/<NUM> and sign language translation <NUM> are broadcasted or streamed to user system(s) 140a-140c/<NUM>, processing hardware <NUM> may execute software code <NUM> to generate sign language translation <NUM> dynamically during the recording, broadcasting, or streaming of primary content <NUM>/<NUM>.

Referring to <FIG> and <FIG>, in yet other implementations in which primary content <NUM>/<NUM> is broadcasted or streamed to user system <NUM>/<NUM>, processing hardware <NUM> of user system <NUM>/<NUM> may execute software code <NUM> to generate sign language translation <NUM> locally on user system <NUM>/<NUM>, and to do so dynamically during play back of primary content <NUM>/<NUM>/<NUM>. Processing hardware <NUM> of user system <NUM>/<NUM> may further execute software code <NUM> to render the performance of sign language translation <NUM> on display <NUM>/<NUM> contemporaneously with rendering primary content <NUM>/<NUM> corresponding to sign language translation <NUM>.

With respect to the method outlined by flowchart <NUM>, it is noted that, in some implementations, actions <NUM>, <NUM>, <NUM>, and <NUM> may be performed in an automated process from which human participation may be omitted.

Claim 1:
A method comprising:
receiving a primary content (<NUM>, <NUM>, <NUM>);
executing at least one of a visual analysis or an audio analysis of the primary content (<NUM>, <NUM>, <NUM>);
generating, based on executing the at least one of the visual analysis or the audio analysis, an accessibility track, wherein the accessibility track comprises:
a sign language performance performed using an animated model and an emotive data set utilized to control the animated model to perform emotions or gestures, wherein the emotions or gestures are performed with at least one of determined speed, forcefulness or emphasis; or
one or more haptic effects configured to be actuated when the primary content (<NUM>, <NUM>, <NUM>) reaches a location corresponding to each of the one or more haptic effects;
synchronizing the accessibility track to the primary content (<NUM>, <NUM>, <NUM>); and
supplementing the primary content (<NUM>, <NUM>, <NUM>) with the accessibility track to provide an accessibility enhanced content (<NUM>, <NUM>, <NUM>).