Data storage server with on-demand media subtitles

A network-attached storage device (NAS) includes a non-volatile memory module storing a media stream, a network interface, and control circuitry coupled to the non-volatile memory module and to the network interface and configured to connect to a client over a network connection using the network interface, receive a request for the media stream from the client, determine subtitle preferences associated with the request for the media stream, access an audio stream associated with the media stream, generate subtitles based on the audio stream, and send a transport stream to the client over the network connection, the transport stream including the media stream and the subtitles.

BACKGROUND

Field

This disclosure relates to data storage systems. More particularly, the disclosure relates to systems and methods for implementing internal on-demand subtitle generation in data storage servers/devices.

Description of Related Art

A storage system can be used to store media content data streams. For example, media content streams can be stored in memory in the Moving Picture Experts Group Transport Stream (MPEG-TS) format. Such media streams can include certain elementary stream constituents, including one or more video streams, audio streams, and possibly subtitle streams. Subtitle elementary streams of native media content streams are generally limited to a finite number of languages and/or types.

DETAILED DESCRIPTION

While certain embodiments are described, these embodiments are presented by way of example only, and are not intended to limit the scope of protection. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the scope of protection.

Overview

With respect to media content consumption by consumers, language limitations of the consumer/user can limit media viewership. Subtitles in media content represent text derived from either a transcript or screenplay of the dialogue or commentary in the media content, such as a film, television program, videogame, and/or other video content. Generally, subtitled may be displayed on/over the bottom of the video frame corresponding to the timestamp of the subtitle. Subtitles can be of the form of written translation of the dialogue in a foreign language or a written rendering of dialogue in the same language, with or without added information to help viewers who are deaf or hard of hearing to follow the dialogue, or people who cannot understand spoken dialogue or have accent-recognition problems.

According to some solutions, subtitles are available as native elementary subtitle streams of certain media content streams. However, such native subtitle streams may be undesirably limited and not flexible in some situations. In particular, such native subtitle streams may not suit all end-users with respect to language and/or other parameter(s). Furthermore, use of client applications that are configured to add subtitles to stored media content streams may require that the whole content of the media stream be retrieved from the relevant data storage server/device and written back to the data storage server/device. Considering that more than 80% of stored data is generally media content, the requirement of transferring media content items back and forth between the client and the data storage server can impose a resource burden and inefficiency with respect to client-side subtitle generation solutions.

Disclosed herein are systems, devices, and methods for implementing in-house/internal subtitle creation and/or integration for/with an internally stored native media content/data stream by a data storage server/device, such as a network-attached storage device (NAS). Such solutions of the present disclosure can allow for on-demand subtitle requests from a client, wherein the resultant transport stream, when decoded, can be played by the playback system along with the newly-generated subtitles received from the data storage server/device in response to the on-demand subtitle request from the client. The resulting transport stream, with the new subtitle file/text, may be playable as specified under the Moving Picture Experts Group (MPEG) broadcast standard. By enabling internal subtitle generation in a data storage device/server, the user experience may be enhanced.

Although certain embodiments are disclosed herein in the context of data storage servers, and in particular network-attached storage devices (NAS) and systems, it should be understood that the principles disclosed herein may be applicable in other data storage environments wherein on-demand subtitle provision is desired.

In some implementations, the present disclosure relates to a network-attached storage device (NAS) comprising a non-volatile memory module storing a media stream, a network interface, and control circuitry coupled to the non-volatile memory module and to the network interface and configured to connect to a client over a network connection using the network interface, receive a request for the media stream from the client, determine subtitle preferences associated with the request for the media stream, access an audio stream associated with the media stream, generate subtitles based on the audio stream, and send a transport stream to the client over the network connection, the transport stream including the media stream and the subtitles.

The control circuitry may be further configured to generate a subtitle file including a payload comprising the subtitles and timestamp data associated with the subtitles. For example, the transport stream can include the subtitle file. In the transport stream, the subtitle file can be separate from the media stream. In some embodiments, in the transport stream, the subtitle file is part of the media stream. The subtitle file can be, for example, a SubRip Subtitle (SRT) file.

In some embodiments, the media stream is stored in a first region of the non-volatile memory module, first region having a first endurance designation, and the control circuitry is further configured to store the subtitle file in a second region of the non-volatile memory module, the second region having a second endurance designation that is different from the first endurance designation. For example, the first region may correspond to a first namespace and the second region may correspond to a second namespace.

The control circuitry may be further configured to append the subtitles to the media stream. In some embodiments, the media stream includes a native subtitle stream. For example, the control circuitry may be further configured to add the subtitles to the native subtitle stream.

In some implementations, the present disclosure relates to a method of generating on-demand subtitles. The method comprises receiving a request for a media stream stored in a non-volatile data store, receiving an on-demand subtitle request associated with the request for the media stream, retrieving an audio stream associated with the media stream from the non-volatile data store, generating subtitles based on the audio stream, and transmitting a transport stream including the media stream and the subtitles.

The method may further comprise generating a new subtitle file including the subtitles and timestamp data associated with the subtitles. For example, the transport stream can include the new subtitle file. The method may further comprise storing the new subtitle file in the non-volatile data store. For example, the new subtitle file can be stored in a separate region of the non-volatile data store than the media stream. In some embodiments, the method further comprises garbage collecting the new subtitle file without garbage collecting the media stream.

In some implementations, the present disclosure relates to a data storage device comprising non-volatile data storage media storing a media stream, means for communicatively coupling to a host system, and controller means coupled to the non-volatile data storage media and to the means for communicatively coupling, wherein the controller means being configured to receive a request for the media stream from the host system using the means for communicatively coupling, receive an on-demand subtitle request associated with the request for the media stream, using a means for transcribing audio data, generate subtitles from an audio stream associated with the media stream, and transmit a transport stream to the host system, the transport stream including the media stream and the subtitles.

The controller means can be further configured to generate a subtitle file including the subtitles and timestamp data associated with the subtitles. In some embodiments, the controller means is further configured to store the subtitle file in a region of the non-volatile data storage media separate from the media stream.

In some embodiments, a network attached storage device and/or client device, may be configured to implement an application programming interface that allows for the requesting and provision of on-demand subtitles for a given media content item. Certain data storage systems configured to provide internal, on-demand subtitle provision in accordance with embodiments of the present disclosure may include a data storage server connected to a client. For example, a data storage system may include a home-storage NAS device, on which a user can store his or her video/audio content and get access to the same using certain software clients provided in connection with the NAS and/or compatible therewith.

Network-attached storage (NAS) drives/systems can provide file-level, or object-level, data storage over a computer network, wherein access to the stored data is accessible to one or more clients. Although certain embodiments are disclosed herein in the context of streams, files, file servers, file systems, and other file-level references, it should be understood that such references, as used herein, may refer to object-level data, or any other type of data structure, depending on the implementation.

A NAS may include hardware, software, or a combination of such elements, configured such that the NAS operates as a media/file server.FIG.1is a diagram illustrating an embodiment of a NAS system100, in which a network-attached storage device (NAS)110is communicatively coupled to one or more client devices over a network120. The NAS110may provide file-based, or object-based, data storage services to devices coupled to the network120. Types of client devices that may have access to the NAS110can include phones137, such as smartphones, cable set-top boxes136, smart TV's135, video game consoles134, laptop computers133, tablet computers132, desktop computers131, wearable computers (not shown) and/or other network-connected computing devices. The network120may be a local area network (LAN), a wide area network (WAN) (e.g., the Internet), or other type of computer network, and the connections between the various client components of the system100and the network120may be either wired or wireless.

While certain embodiments are described herein in the context of NAS devices/systems, it should be understood that references herein to NAS's may refer to other types of data storage devices/systems, such as any type of computer device implementing software allowing for data storage access over a network or other connection. Furthermore, some embodiments disclosed herein may be implemented using data storage device connections that are not over a network, but rather a direct client/server connection.

In certain embodiments, the NAS110may be configurable over the network120by a client device interface, such as a web browser of a mobile or desktop computing device. An operating system (e.g., a relatively low-level operating system, such as FreeNAS) may be implemented in the NAS110by a control circuitry130thereof. The NAS110may provide access to files142, such as video files, using one or more network file-sharing protocols, such as Network File System (NFS), Server Message Block (SMB/CIFS), Apple Filing Protocol (AFP), or the like. The NAS110may comprise a data store140for storing the user data (e.g., media streams)142, as well as certain metadata144, such as system tables or the like, and/or other types of data. The data store140may comprise one or more non-volatile memory devices or modules and may include any type of data storage media (e.g., solid-state, magnetic).

Although certain embodiments are disclosed herein in the context of video content streams and video content, it should be understood that principles disclosed herein may be applicable to other types of data containers or content streams. Storage access clients may be mobile clients, desktop clients, and/or web clients. Among other types of user content, users may store video content of various kinds on a NAS. It may be desirable for users, with the assistance of certain client applications, to have the ability to play videos stored on a NAS. That is, the NAS may advantageously provide video playback to client devices as an integral feature of the device. The terms “video content” and “content” are used here and according to their broad and/ordinary meanings and may be used to describe a video file or container, and/or video stream content contained within a file or container.

The stored user data may include, for example, audio, video, still images, voice, text, or other form of media. Media content stored in the data store140, such as media streams142, may be embodied in containers, and may be in a single uncompressed container format for each media type. For example, audio files may be archived in data storage14as PCM (pulse code modulation) format, .wav or .aiff files, or as BWF (broadcasting wave format). Still images may be stored in FITS (Flexible Image Transport System) or TIFF (Tagged Image File Format). Video may be stored in Audio Video Interleave (AVI), ANIM, Advanced Systems Format (ASF), DVR-MS, Interchange File Format (IFF), QuickTime multimedia file format (MOV), MPEG-1, MPEG-2, MP4 or the like.

The NAS110may be configured to implement encryption for the media streams142stored in the data store140. For example, the NAS110may implement Transport Layer Security (TLS), Secure Sockets Layer (SSL), and/or Advanced Encryption Standard (AES) keys (e.g., 256-bit, 128-bit, etc.) to protect files in rest and/or in motion. The NAS110may further be configured to implement one or more additional security features, such as user verification, forward secrecy, and/or the like.

The NAS110includes a media server138configured to serve media stream content142, such as transcoded video file content, to clients connected thereto and authorized to use such content. With respect to playback of the media content142stored in the NAS110, certain challenges may be associated with playback by a particular user of such content. For example, the media streams142may comprise various video file containers and/or codecs. The term “codec” is used herein and according to its broad and/ordinary meaning, and may be used to describe a device or computer program for encoding or decoding a digital data stream or signal, such as a video stream, audio stream, or the like. The control circuitry130of the NAS110may implement one or more video and/or audio codecs to encode data streams for transmission to the various clients, possibly in encrypted form. The clients may comprise decoder functionality for reversing the encoding for playback or editing. The term “control circuitry” is used herein according to its broad and ordinary meaning, and may refer to any collection of processors, processing circuitry, processing modules/units, chips, dies (e.g., semiconductor dies including come or more active and/or passive devices and/or connectivity circuitry), microprocessors, micro-controllers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuitry, analog circuitry, digital circuitry, and/or any device that manipulates signals (analog and/or digital) based on hard coding of the circuitry and/or operational instructions. Control circuitry referenced herein may further comprise one or more, storage devices, which may be embodied in a single memory device, a plurality of memory devices, and/or embedded circuitry of a device. Such data storage may comprise read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and/or any device that stores digital information. It should be noted that in embodiments in which control circuitry comprises a hardware and/or software state machine, analog circuitry, digital circuitry, and/or logic circuitry, data storage device(s)/register(s) storing any associated operational instructions may be embedded within, or external to, the circuitry comprising the state machine, analog circuitry, digital circuitry, and/or logic circuitry. Any types of control circuitry or controllers disclosed herein may be considered types of controller means that may be implemented to perform certain functionality associated with embodiments of the present disclosure.

Host/Storage Server Configuration

Storage systems suitable for use in implementing aspects of embodiments of the present disclosure are shown inFIGS.2A-2C.FIG.2Ais a block diagram illustrating a non-volatile storage system205according to one or more embodiments of the present disclosure. Referring toFIG.2A, the non-volatile storage system205includes a controller102and non-volatile media204that may be made up of one or more non-volatile memory die. As used herein, the term die refers to the collection of non-volatile memory cells, and associated circuitry (e.g., control circuitry) for managing the physical operation of the non-volatile memory cells, that are formed on a single semiconductor substrate. The control circuitry202, which may be considered, and is referred to herein as such for convenience as an example implementation, a “controller” as understood by those having ordinary skill in the art and described in certain detail below, interfaces with a host system201and transmits command sequences for read, program, and erase operations to non-volatile data storage media204.

The controller202(which may be a non-volatile memory controller (e.g., a flash, resistive random-access memory (ReRAM), phase-change memory (PCM), or magnetoresistive random-access memory (MRAM) controller)) can take the form of processing circuitry, a microprocessor or processor, and a computer-readable medium that stores computer-readable program code (e.g., firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, for example. The controller202can be configured with hardware and/or firmware to perform the various functions described below and shown in the flow diagrams. Also, some of the components described as being internal to the controller can also be stored external to the controller, and other components can be used. Additionally, the phrase “communicatively coupled” with/to could mean directly in communication with or indirectly (wired or wireless) in communication with through one or more components, which may or may not be shown or described herein.

As used herein, a non-volatile memory controller is a device and/or control circuitry that manages data stored in non-volatile media and is configured to communicate with a host, such as a computer or electronic device. A non-volatile memory controller can have various functionality in addition to the specific functionality described herein. For example, non-volatile memory controllers disclosed herein can be configured to format communicatively coupled non-volatile media to ensure the memory is operating properly, map out bad non-volatile memory cells, and allocate spare cells to be substituted for future failed cells. Some part of the spare cells can be used to hold firmware to operate the non-volatile memory controller and implement other features. In operation, when a host needs to read data from, or write data to, the non-volatile memory, it can communicate with the non-volatile memory controller. If the host provides a logical address to which data is to be read/written, the non-volatile memory controller can convert the logical address received from the host to a physical address in the non-volatile memory. (Alternatively, the host can provide the physical address.) The non-volatile memory controller can also perform various memory management functions, such as, but not limited to, wear leveling (distributing Writes to avoid wearing out specific blocks of memory that would otherwise be repeatedly written to) and garbage collection (after a block is full, moving only the valid pages of data to a new block, so the full block can be erased and reused).

The non-volatile media204may include any suitable non-volatile storage medium, including resistive random-access memory (ReRAM), magnetoresistive random-access memory (MRAM), phase-change memory (PCM), NAND flash memory cells and/or NOR flash memory cells. The memory cells can take the form of solid-state (e.g., flash) memory cells and can be one-time programmable, few-time programmable, or many-time programmable. The memory cells can also be single-level cells (SLC), multiple-level cells (MLC), triple-level cells (TLC), or use other memory cell level technologies, now known or later developed. Also, the memory cells can be fabricated in a two-dimensional or three-dimensional fashion.

The interface between the controller202and non-volatile media204may be any suitable flash interface, such as Toggle Mode200,400, or800. In some embodiments, the storage system205may be a card-based system, such as a secure digital (SD) or a micro secure digital (micro-SD) card. In an alternate embodiment, storage system205may be part of an embedded storage system.

The non-volatile storage system205(sometimes referred to herein as a storage module) can include a single channel between the controller202and the non-volatile media204. However, the subject matter described herein is not limited to having a single memory channel. For example, in some storage system architectures (such as the ones shown inFIGS.2B and2C), 2, 4, 8 or more memory channels may exist between a controller and non-volatile media, depending on controller capabilities. In any of the embodiments described herein, more than a single channel may exist between the controller and the non-volatile media, even if a single channel is shown in the drawings. InFIGS.2A-2C, certain reference numbers are reused as a matter of convenience for devices and modules having features that are similar in one or more respects. However, as with any of the embodiments disclosed herein, re-use of common reference numbers in the drawings does not necessarily indicate that such features, devices, components, or modules are identical.

FIG.2Billustrates a storage module200that includes a plurality of non-volatile storage systems205. As such, storage module200may include a storage controller212that interfaces with a host201and with the storage module200, which includes a plurality of non-volatile storage systems205. The interface between storage controller212and the non-volatile storage systems205may be a bus interface, such as a serial advanced technology attachment (SATA), peripheral component interconnect express (PCIe) interface, or double-data-rate (DDR) interface. In some embodiments, the storage module200may be a solid state drive (SSD), or non-volatile dual in-line memory module (NVDIMM), such as may be found in server PC or portable computing devices, such as laptop computers, and tablet computers.

FIG.2Cis a block diagram illustrating a hierarchical storage system250. The hierarchical storage system250includes a plurality of storage controllers212, each of which controls a respective storage system/module200. Host systems201may access data storage devices within the storage system via a bus interface. In some embodiments, the bus interface may be a Non-Volatile Memory Express (NVMe) or Fibre Channel over Ethernet (FCoE) interface. In some embodiments, the system illustrated inFIG.2Cmay be a rack-mountable mass storage system that is accessible by multiple host computers, such as may be found in a data center or other locations where mass storage is needed.

FIG.3is a block diagram illustrating components of a controller102in more detail. The controller102includes a front end module108that is configured to interface with a host (not shown), a back end module170that is configured to interface with non-volatile media104(e.g., one or more non-volatile memory die), and various other modules/circuitry that are configured to perform various functions. Control circuitry “modules” described herein may take the form of a packaged functional hardware unit designed for use with other components, a portion of a program code (e.g., software or firmware) executable by a (micro)processor or processing circuitry that performs a particular function, or a self-contained hardware or software component that interfaces with a larger system, for example. Modules of the controller102may include a video frame segregator111, which can be implemented in hardware or software/firmware to extract various video frames from a video stream, wherein subtitles can be superimposed on such video frames in connection with some embodiments disclosed herein. The video frame segregator111can be configured to perform certain of the algorithms and methods discussed below and shown in the attached drawings.

Referring again to modules of the controller102, a buffer manager/bus controller114configured to manage buffers in random access memory (RAM)116and control the internal bus arbitration of the controller102. A read only memory (ROM)118may be includes that stores system boot code. Although illustrated inFIG.3as located separately from the controller102, in other embodiments one or both of the RAM116and ROM118may be located within the controller102. In yet other embodiments, portions of RAM and ROM may be located both within the controller102and outside the controller.

The front end module108can include a host interface120and a physical layer interface (PHY)122that provide the electrical interface with the host or next level storage controller. The choice of the type of host interface120can depend on the type of memory being used. Examples of host interfaces120include, but are not limited to, SATA, SATA Express, serially attached small computer system interface (SAS), Fibre Channel, universal serial bus (USB), PCIe, and NVMe. The host interface120typically facilitates transfer for data, control signals, and timing signals.

The back end module170may include an error correction code (ECC) engine124that is configured to encode data bytes received from the host, and decodes and error-correct data bytes read from the non-volatile media104. A command sequencer126may be included that is configured to generate command sequences, such as program and erase command sequences, to be transmitted to the non-volatile media104. In some embodiments, a RAID (Redundant Array of Independent Drives) module128is included that is configured to manage generation of RAID parity and recovery of failed data. The RAID parity may be used as an additional level of integrity protection for the data being written into the media104. In some cases, the RAID module128may be a part of the ECC engine124. A memory interface180may be included that is configured to provide the command sequences to non-volatile media104and receive status information from the non-volatile media104. In some embodiments, the memory interface180may be a double data rate (DDR) interface, such as a Toggle Mode200,400, or800interface. A flash control layer132may be included that is configured to control the overall operation of back end module170.

The storage system100can also include certain other discrete components160, such as external electrical interfaces, external RAM, resistors, capacitors, or other components that may interface with controller102. In some embodiments, one or more of the physical layer interface122, RAID module128, media management layer138and buffer management/bus controller114can be considered optional components that are not necessary in the controller102.

FIG.4is a block diagram illustrating components of the non-volatile media (e.g., non-volatile memory die) in more detail. The non-volatile media104can include certain peripheral circuitry141and one or more non-volatile memory arrays162. The non-volatile memory array162can include non-volatile memory cells used to store data. The non-volatile memory cells may be any suitable non-volatile memory cells, including ReRAM, MRAM, PCM, NAND flash memory cells and/or NOR flash memory cells in a two dimensional and/or three dimensional configuration. The non-volatile media104can further include a data cache156that is configured to cache data written and/or read to/from the non-volatile memory array162. The peripheral circuitry141can include a state machine152that provides status information to the controller102.

Returning again toFIG.3, the flash control layer132(which may be referred to herein as the flash translation layer (FTL) or, more generally, the “media management layer,” as the memory may not be flash) can be configured to handle flash errors and/or interface with the host. In particular, the FTL, which may be implemented in firmware code, may be responsible for the internals of memory management and/or configured to translate writes from the host into writes to the non-volatile media104. The FTL may be needed due to limited endurance characteristics of the media104, wherein the media104may be configured to only be written in multiples of pages, and/or may not be written unless it is erased together as a block or other areas or region of the media. These limitations of the media104may not be visible to the host. Accordingly, the FTL can be configured to translate writes from host into writes to the memory104.

The FTL may include a logical-to-physical (L2P) address map and allotted cache memory. In this way, the FTL can be configured to translate logical block addresses (“LBAs”) from the host to physical addresses in the media104. The FTL can include other features, such as, but not limited to, power-off recovery (so that the data structures of the FTL can be recovered in the event of a sudden power loss) and wear leveling (so that the wear across memory blocks is even to prevent certain blocks from excessive wear, which would result in a greater chance of failure).

On-Demand Subtitle Provision by Data Storage Device/Server

FIG.5is a block diagram illustrating a data storage server system500, such as a network-attached storage (NAS) system, according to an example embodiment. The system500includes a data storage server550, which may represent an embodiment of the NAS110ofFIG.1. Although the storage server550may represent any type of data storage server device or system, the storage server550may be described below in certain contexts as a NAS for illustration purposes. The data storage server (e.g., NAS)550may include certain control circuitry or components configured to implement a media server552. For example, the illustrated embodiment ofFIG.2comprises control circuitry functional modules and components including the file server552, client application interface551, subtitle management circuitry553, and file system557. The control circuitry of the various illustrated functional module may be implemented as a combination of one or more processors, chips/dies, field-programmable gate arrays (FPGA), data and/or power transmission channels or busses, volatile and/or non-volatile memory modules, stored code, and/or other components. Furthermore, although the control circuitry of the data storage server550is illustrated as various separate modules, it should be understood that the functionality represented thereby may be implemented using any configuration of modules or control circuitry.

The data storage server550includes non-volatile data storage540. The data storage540may comprise one or more disks, wherein the NAS550further comprises one or more heads actuated radially over the disk(s), and a spindle motor for rotating the disk(s). Alternatively or in addition to magnetic (or optical) rotating media, the non-volatile data storage540may comprise solid-state memory and/or other non-volatile memory. In certain embodiments, the data storage server550may comprise one or more hybrid hard drives including both magnetic media and solid-state media. In addition to the illustrated modules and components, the data storage server550may further include one or more additional network interfaces, processors, data and/or power communication buses, memories, boards, chips/dies, or the like.

Media requests and transport streams may be communicated between the client530and the data storage server550over any type of wired or wireless connection (e.g., network connection). Such connection may comprise any digital or other communications network capable of transmitting messages between senders and receivers, such as any number of public or private data connections, links or networks supporting any number of communications protocols. In some embodiments, the client530and server550are configured to communicate over the Internet, for example, or any other network based upon TCP/IP or other conventional protocols, and/or over a wireless and/or wired telephone network, such as a cellular communications network for communicating with mobile phones, personal digital assistants, and/or the like. The client530and server550may further communicate over any sort of wireless or wired local area networks, such as one or more IEEE 802.3 and/or IEEE 802.11 networks.

The data storage server550may comprise a NAS that may be, for example, a personal in-home box, which may be accessible by the client530either locally (e.g., over a LAN connection) or through a cloud-based connection. The client530may be configured to implement a server interface application536configured to communicate with the data storage server550according to a particular application programming interface (API). For embodiments in which the client530is a mobile computing device (e.g., smartphone), the server interface application536may be a mobile client application. The server interface application536may be configured to make media content item access requests incorporating various parameters, including subtitle preference parameters, as described herein. Where the client530is communicatively coupled to the data storage server550over a LAN connection, the client530may be configured to search for data storage server devices on a network, wherein such search may produce a list of all available devices based on, for example, IP address.

The data and/or requests communicated between the client530and the data storage server550may be implemented through a particular communication protocol that both the server interface application536of the client530and the client application interface551of the data storage server550are designed to execute. For example, in some embodiments, the client530and data storage server550communicate according to a representational state transfer (REST) application programming interface (API), or other stateless interface, which may provide desirable interoperability between the system components. The implemented API may allow for clients to utilize the file system557of the data storage server550by requesting files as network resources identified by, for example, a network address (e.g., Uniform Resource Locator (URL), Uniform Resource Identifier (URI), or the like). The requests communicated by the client530to the data storage server550may comprise, for example, HTTP requests (e.g., HTTP 1.1, HTTP/2). The media server/controller552may receive data and storage access commands using the client application interface551, which may be configured to communicate with the client530according to the relevant API (e.g., REST API). In certain embodiments, the client530utilizes a DNS server in communicating with the data storage server550; the data storage server550may be callable through a web address URL (Uniform Resource Locator).

The client530may comprise control circuitry539configured to implement the functionality of the illustrated modules/components. In certain embodiments, the client530is configured to implement a virtual file system (not shown). In some embodiments, the connection between the client530and the data storage server550may be wired, such as through Ethernet, USB, or other connection, or may be wireless, such as through WiFi, Bluetooth, or other wireless connection. In certain embodiments, the connection between the client530and the data storage server550is achieved over the Internet, wherein each of the client530and the data storage server550is connected to the Internet over a wired or wireless connection.

The data storage server550may be configured to implement data redundancy, wherein copies or portions of user data stored in the data storage540are maintained in one or more internal and/or external drives. For example, the data storage server550may implement redundant array of independent disks (RAID) technology, wherein the non-volatile data storage540includes a plurality of internal drives, disks, or other data storage partitions combined into a logical unit for the purposes of data redundancy and performance improvement. In addition, or alternatively, the data storage server550may be configured to implement RAID using one or more internal memory modules in combination with one or more external memory devices. Furthermore, data may be distributed across the RAID memory modules/drives according to any desirable or practical RAID level, depending on the level of redundancy and/or performance desired. For example, the data storage server550may be configured to implement RAID0, RAID1, RAID5, RAID10, or other RAID technology, depending on data reliability, availability, performance and/or capacity considerations or requirements.

The data storage server550may implement a file system557that may be accessible by the client530through the server interface application536for browsing and/or searching. For example, the non-volatile data storage540may comprise some number of fixed-size data segments of data storage (e.g., blocks). The non-volatile data storage540may be configured to process relatively simple data access commands, wherein such commands are received from the file server550over a communication interface (e.g., Integrated Drive Electronics (IDE), Small Computer System Interface (SCSI), Serial ATA (SATA), or the like). The file system557may implement media streams, files, file directories, and/or other data structures that represent data stored in the non-volatile data storage540. In certain embodiments, the file system557maintains certain metadata (e.g., tables) for associating logical files with physical block numbers, or the like. For example, the file data554may comprise a file allocation table (FAT) that describes file/block associations. The file data554may further track unused blocks and/or allocate new blocks, as needed. In certain embodiments, the media server552is configured to provide a file list to the client530for representing to the client530the accessible media content items of file system that are available to the client530.

The client530may comprise a server, a desktop, a laptop, a tablet, a handheld device, or the like, and includes the control circuitry539, which may comprise one or more central processing units (CPUs), memory/data storage devices or modules, network interfaces, and/or input/output interface components, and the like. The control circuitry539of the client530may be configured to execute certain software applications for implementing the functionality described herein, such as the media player532. The media player532may comprise applications that may be executable within an operating system (OS) implemented by the control circuitry539of the client530. The client may one or more local storage devices (not shown), such as hard disks, flash memory modules, solid state disks, optical disks, and the like. In certain embodiments, the client530comprises a network interface (not shown), which may include one or more network adapters (e.g., network interface cards (NICs)), for connecting to the server550over a network.

The client application interface551may be configured to implement various API operations (e.g., REST API operations) that can be invoked by the client530through communication with the data storage server550. API operations can include, for example, creation of, and/or access to, video files maintained in the non-volatile data storage540. Each API operation can be any type of service, application, or the like that can be executed by the data storage server550on behalf of the client530.

The media player532may be configured to send requests, as translated by the server interface application536for communication according to the relevant API, to the data storage server550. The responses from the data storage server550may include media transport streams that may be translated by the server interface application536according to the API and passed to the media player532. Storage access commands communicated by the server interface application536may include media content item access commands issued by the media player532. The video access commands may specify a media content item (e.g., video file/stream) address/location and an on-demand subtitle parameter.

The media player532may comprise a standard media player, and may be configured to request video data using a relevant network application protocol, such as hypertext transfer protocol (HTTP), or the like, by specifying a location (e.g., uniform resource locator (URL)) of the relevant media content item. In some implementations, the media player532may be a web-browser media player, which may be configured to use a video element (e.g., HyperText Markup Language (HTML), HTML5) associated with the browser to process and/or present video content. In some implementations, the media player532may be a mobile application for a mobile device client, such a smart phone. The media player532may be designed to accept a resource location (e.g., URL) that points to a media stream stored at the data storage server550, and to play such stream.

The media player532may comprise one or more codecs533, which may be configured to decode (and/or encode) media (e.g., video and/or audio) content, such as for presentation using a video interface575on a display570associated with the client530. The codec533may be designed to decode data compressed in a particular format, such as H.264, H.265, MPEG-4, or the like.

The client530and/or other client/user may maintain video content on the non-volatile data storage540, among possibly other types of data (e.g., audio, text, etc.). One characteristic of video files that may be relevant in certain environments is the relatively large size of some video files relative to other types of files. In view of such size, may be difficult or undesirable to upload the entire video file to the client530prior to initiating playback thereof. Therefore, streaming of video content may be desirable in order to improve the playback experience of the client530. The video file541may be stored in the non-volatile data storage540in any suitable or desirable manner. For example, a user/client may provide the file for storage over a wired (e.g., USB) local connection, or over a remote or local wireless connection.

In some embodiments, computer-readable program code/media that, when executed using one or more processors of the control circuitry539, causes the control circuitry539to implement the functionality of the media player532is stored in one or more data storage devices of the control circuitry539of the client530. In some embodiments, functionality of the media player532is implemented exclusively in hardware.

The storage server550can be used to store a data stream sent to it by a host, such as the client230or other client/host. For example, the data storage server550can be configured to receive the media stream580and store the stream in the non-volatile data storage540. In some embodiments, the data/stream580can be stored in the Moving Picture Experts Group Transport Stream (MPEG-TS) format. For example, the entire stream580can be stored as a single entity, which can be advantageous due to the relatively easy storage and retrieval thereof. However, such a process generally results in uniform data characteristics (e.g., endurance) across the entire spectrum of data of the stream580. Therefore, storage of one or more elementary streams associated with the media stream580, such as the new subtitle file/stream590, in a separate block/region of the non-volatile data storage540can be advantageous. For example, with respect to the subtitle file/stream590, such portion of the data associated with the media stream580may only be required to be stored for a period of time. According to some implementations in which the new subtitle data/stream is stored together with the media stream580(e.g., as a subtitle elementary stream588or superimposed on the frames of the video stream582), in order to remove the new subtitle file/stream590after it becomes unnecessary, it may be necessary for the entire data stream580to be read from memory, wherein the unnecessary video frames and/or elementary stream(s) is/are removed, and the resulting new stream is stored back in memory, which may be an inefficient process that reduces performance and increases write amplification. Further, the frames/stream(s) that will be removed are stored with the same endurance and protection parameters as those that will be kept. The same issue can be present with different packet identifiers.

In some embodiments, the subtitle file/data590is stored in the memory540of the storage system/server550using different storage options relative to the media stream580. This addresses the problem mentioned above where storing the MPEG-TS data as a single stream with the new subtitle data creates throughput issues when the data is phased out after a period of time. Such separate storage can also be performed on different elementary streams or packet identifiers (PIDs), which are the data that identifies each table or elementary stream in a transport stream.

The following paragraphs provide examples in which the data stream is in the MPEG transport stream (MPEG-TS, MTS, or simply “transport stream”) format specified in the MPEG-2 protocol. It should be understood that this is merely an example, and other types of formats, frame types, and storage options can be used. MPEG-TS is a standard digital container format for transmission and storage of audio, video, and Program and System Information Protocol (PSIP) data. It is used in broadcast systems such as Digital Video Broadcasting (DVB), Advanced Television Systems Committee (ATSC) and Internet Protocol television (IPTV). Generally, a transport stream is a container format encapsulating packetized elementary streams, which may include error-correction and synchronization pattern features for maintaining transmission integrity when the communication channel carrying the stream is degraded. References herein to transport streams may refer to program streams, which streams designed for reasonably reliable media, such as discs (like DVDs), whereas transport streams are designed for less reliable transmission, namely terrestrial or satellite broadcast. A transport stream may carry multiple programs.

Transport streams, like the transport stream585transmitted from the data storage server550to the client530including the on-demand subtitle data/file590, can encapsulate a number of substreams, such as packetized elementary streams (PESs) which in turn wrap the main data stream using the MPEG codec or any number of non-MPEG codecs (such as Dolby Digital (AC-3) or Digital Theater Systems (DTS) audio, and Motion JPEG (MJPEG) or JPEG 2000 video), text and pictures for subtitles, tables identifying the streams, and even broadcaster-specific information such as an electronic program guide. Many streams can be mixed together in some cases. Generally, each transport stream may be comprised of (at most) 188-byte sections that are interleaved together. For example, the transport stream585can comprise a plurality of sequenced network packets. Such packets can include a sync byte and a header, that may be followed with optional additional headers, wherein the rest of the packet consists of payload data. The packets may be up to 188 bytes in length, but the communication medium may add additional information. The packet payload can include header and payload data for various packetized elementary streams (PES's), such as the video stream582, audio stream583, native subtitle stream586, and/or new subtitle stream588.

With further reference to the media stream580, depending on the type of container, the video stream582may be represented in any of a variety of possible formats. For example, the media stream580may comprise a single video stream, an advanced audio coding (AAC) audio stream and/or a dedicated-to-sound (DTS) audio stream, as well as subtitle streams in one or more languages. The video stream component of the client codec533and/or decoder557of the data storage server550codec may be any suitable or desirable codec, such as H.264, or the like. The audio stream component of such modules may be MP3, AAC, AC-3, or the like.

The user I/O module537may provide an interface for communicating with a user using one or more I/O components, such as the display570and/or one or more user input device interfaces (e.g., mouse, keyboard, touchscreen, or the like). The user I/O module537may provide the video interface575for viewing, by the user, certain video content and/or subtitles599(e.g., on-demand subtitles).

The media player532may be configured to display the new subtitles599in the video interface575by displaying video frames that have the new subtitles superimposed thereon. For example, the data storage server550may provide the video stream with video frames having the new subtitles superimposed thereon. Alternatively, the media player532may be configured to display the new subtitles599by reading and displaying the subtitle text592in a time-synchronized fashion in the video interface575over the respective video frames.

The transcriber555of the data storage server550may be configured to implement conversion of audio data to text data. The transcriber555may comprise hardware for hardware transcribing, which may free-up processor and/or other resources relative to exclusively-software-implemented transcribing embodiments. In certain embodiments, the transcriber555may comprise hardware that is a component of a system on a chip (SOC) of the data storage server550, wherein the SOC comprises some or all of the control circuitry of the media server552. In certain embodiments, the transcriber555comprises software code/firmware for implementing transcribing functionality and/or interacting with hardware transcoder component(s).

With respect to on-demand subtitle generation, the client530may provide a request including one or more of a file name or other identifier, desired video content resolution, and/or desired bit rate, as well as possibly including on-demand subtitle preference/request, over the connection between the client530and the file server552. The file server552may provide the audio stream583of the requested media stream580to the subtitle management circuitry553, which may produce an output stream585and provide the same to the client application interface551, which forwards the responsive transport stream585to the client530in accordance with the relevant API. In certain embodiments, the media server552provides the newly-generated subtitle text together with the associated video stream frames dynamically as the bytes of the subtitle text are generated by the transcriber555. Therefore, if a user provides seeking input while playing the media stream, the new subtitles associated with the requested byte may not have been transcribed by the transcriber555at that point in time. Therefore, such seeking may introduce some amount of latency in some implementations to allow the subtitle manager553time to generate the desired subtitles.

The client530(e.g. media player532) may be configured to receive and/or provide user preference data relating to on-demand subtitle preferences/settings requested in connection with a particular media content item, such as the media stream580stored in the nonvolatile data storage540of the data storage server550. The media server/controller552may be configured to decode an audio stream or other audio data583of the requested media stream580, which may be present in, for example, an audio PID/stream of the media stream580, wherein the media server/controller552of the data storage device/server550is configured to internally perform audio-to-text conversion from the decoded audio data using the transcriber/speech-recognition circuitry555. The transcriber555may use any suitable or desirable speech recognition model. Such speech recognition model may be updated when required, such as by the client530and/or media server552, to perform desired speech-to-text functionality.

The media server/controller552may be configured to use the resulting subtitle text generated by the transcriber circuitry555to create a separate/new subtitle file590, which may be embodied in any suitable or desirable data structure or protocol. For example, subtitle files generated in connection with embodiments of the present disclosure may comprise SubRip subtitle (SRT) files. The subtitle file590generated by the media server/controller552may be appended to the stored native media stream580, and/or may be stored separately in a separate block or group of the nonvolatile data storage540from the native media stream580. The new subtitle stream588is illustrated as an example appended subtitle stream appended to the native media stream580. The new subtitle stream588may be appended in addition to, or in place of, one or more native subtitle streams586, which may be pre-generated subtitle streams provided and stored with the native media stream580prior to the on-demand subtitle request from the client530.

The file server552, and in particular the subtitle management553portion of the control circuitry of the file server552, includes decoder circuitry557. The decoder circuitry557may be configured to decode audio and/or video data, such as the audio stream583of the native media transport stream580stored in the nonvolatile data storage540. The subtitle management circuitry553further comprises the transcriber circuitry555, which may be configured to receive as input audio data, such as the audio stream583of the native media transport stream580and convert the audio data to text data. That is, the transcriber555may provide speech-to-text conversion/transcription functionality. The media server552may be configured to retrieve the audio stream583and transcribe the audio stream to generate the subtitle text data592.

The data storage server550can receive an on-demand subtitle request with necessary configuration data from the client530. In turn, the data storage server550can decode the audio stream583of the corresponding stored transport media580, which may be used as input to the transcriber module555.

The subtitle manager circuitry553may be configured to generate new subtitle files, such as the new subtitle file590, based at least in part on certain data stream(s) (e.g., elementary data stream(s)) of a native media transport stream. The media server552may receive a command from the client532to access the native media stream580, which may represent a video file and/or other media file. The media server252may further receive, as part of the media request/access command, or as part of a separate command from the client530, preference data for on-demand subtitles of a specified language and/or having certain parameters. In response, the media server552may retrieve the native media stream580from the nonvolatile data storage540and extract/retrieve the audio stream583therefrom. The audio stream583may be provided to the transcriber circuitry555to generate subtitle text corresponding to the audio data of the audio stream583. In order to transcribe the audio data, the subtitle manager553may first decode the audio stream583using the decoder circuitry557. The subtitle management circuitry553may further be configured to generate the new subtitle file590based at least in part on the subtitle text data generated by the transcriber555. For example, the subtitle manager553may construct the new subtitle file590according to the SubRip Subtitle (SRT) standard. The new subtitle file590may include a header593, as well as a payload including the subtitle text592and certain timestamp data591relevant for synchronizing the subtitle data592with video frames of the video stream582. In some embodiments, the media stream580includes timestamp/synchronization data for synchronizing the frames of the video stream582with the audio stream583data; the timestamp data591of the new subtitle file590can indicate temporal relationships with the timestamp/synchronization data indicating temporal relationships between the video stream582and the audio stream. The video stream may be time-synched to the audio stream to provide lip synchronization. The timestamp data591may likewise synchronize the subtitle data592to the video frames of the video stream582the video itself is synchronized to audio for lip synchronization as following—relevant for synchronizing the subtitle data592with video frames of the video stream582, the video itself is typically synchronized to audio frames

Use of SRT file standard may be desirable to provide Moving Picture Experts Group (MPEG) compatibility. The newly-created subtitle file590can be stored in the nonvolatile data storage540and integrated into the native media stream580. Generally, the new subtitle file590, such as an SRT file, may comprise the subtitle text592and the corresponding timestamp data591for audio presentation as the payload of the file590. The new subtitle file590can be added to the existing elementary stream or treated as a separate program, which the graphical layer of the media player532can superimpose on the respective video frames. That is, the subtitle management circuitry553may be configured to superimpose the newly-generated subtitle text onto the video stream582of the native media stream580, such that video frames thereof include new superimposed subtitle text corresponding to the video frame timestamp.

After generating the new subtitle file590, the media server552may be configured to append the subtitle file590and or subtitle text592to the native media stream580, as indicated by the optional presence of the new subtitle stream588as part of the media stream580. In some embodiments, the media server552may store the new subtitle file590separately from the native media stream580. For example, the new subtitle file590may be stored in a separate area, region, block, and/or other type of partition from the native media stream580.

Storage of the new subtitle file590separately from the native media stream580can provide certain technological benefits and advantages. For example, according to some implementations, the new subtitle file590may be stored in a temporary storage area that may be garbage-collected separately from the region548of data storage in which the native media stream580is stored. For example, in some implementations, the subtitle file590may be saved in a separate flash block from the native media stream580, thereby allowing for garbage collection of the flash block546in which the new subtitle file590is stored separately from the flash block548in which the native media stream580is stored. That is, the native media stream580and the new subtitle file590may be associated with blocks or other partitions of data storage media that have different endurance characteristics. Therefore, if the new subtitle file590is no longer needed, erasure of the block of storage546storing the new subtitle file590may be performed, thereby freeing up data storage space for other uses, without erasing or garbage-collecting the block548storing the native media stream580.

Generally, it may be undesirable to store subtitle data in a separate block/region of storage from a media stream with which it is associated, as such implementation would require a separate retrieval of the subtitle data in addition to the retrieval of the media stream, as well as maintenance of mapping information indicating the connection between the media and subtitle files, thereby potentially negatively impacting media access/provision efficiency. However, embodiments of the present disclosure may be implemented in spite of such inefficiencies in order to improve efficiency relating to garbage collection, wear leveling, and/or other data storage maintenance operation(s). For example, garbage collection technology improvement through the use of separately-stored on-demand subtitle files can provide improved operational efficiency and data storage availability for the data storage server550, such as by freeing up data storage space. Garbage collection generally involves moving certain valid data from various blocks/regions of the non-volatile data storage540to purge such blocks of valid data and thereby allow for the erasure/reuse of the blocks. This is due to the necessity of blocks of, for example, NAND flash memory to be erased in their entirety. When the new subtitle file590is stored in another block of the non-volatile data storage540separate from the native media stream580, if the subtitle file590becomes no longer needed, the block in which the subtitle file is stored can be erased once any other valid data in the block has been moved, without the necessity of moving the new subtitle file590or the native media stream580. Alternatively, if the new subtitle stream588were stored as an elementary stream of the native media stream580, together in the block/region548of the data storage in which the media stream580is stored, it may not be practical or desirable to erase the block/region548once the subtitles become outdated or stale without transferring data of the native media stream580to another block/region. Therefore, embodiments of the present disclosure can reduce the workload of the data storage server550.

At the client530, after receiving the media stream580and the new subtitle file590, which may be transmitted separately, or as a common transport stream585, the media player application532may decode the media stream580and the new subtitle file590and display video frames573associated therewith in a video interface window575. The media player532may further display the new subtitles599based on the subtitle text592in the video interface575. In some embodiments, the new subtitles599may be part of the decoded video stream received by the client530or may be added by the media player532and superimposed over the video frame573using the new subtitle file590. The timestamp data591may be used to govern when certain text of the subtitle text592is displayed to ensure that the subtitles correspond to the correct times/frame of the video media.

The data storage server/device550can send the resulting transport stream585on-demand to the client530, wherein the transport stream585includes the newly-generated subtitle file590and/or subtitle text592. In some implementations, the subtitle text592may be superimposed on video frames of the video stream582of the media stream580in addition to, or rather than, sending as a separate subtitle file. That is, the video frames of the video stream582may have superimposed thereon relevant lines of text of subtitle corresponding thereto in a time-synced manner. The provision by the data storage server550of on-demand subtitle content can advantageously enhance the user experience by increasing the subtitle viewing options for a media content item. On-demand subtitle provision and generation in accordance with embodiments of the present disclosure further provides technological improvement to media storage, request, and presentation technologies, as described in detail herein.

As shown, the data storage server550is configured to internally generate the on-demand subtitles and provide the same together with the media stream580, thereby obviating the need to transfer the media stream580to the client530prior to the generation of the on-demand subtitles.

Once the text subtitles have been generated by the subtitle manager653, the subtitle manager653may be configured to generate a subtitle file incorporating the subtitle text, as well as certain timestamp and/or other temporal data. The media may be attached to the subtitle text or file in any suitable or desirable manner. For example, the native/legacy media stream580includes one or more video582and audio583streams. When the new subtitle text/stream588is created, it is desirable for the subtitles to be temporally synchronized with the video and audio streams. The new subtitle stream588includes timestamp data allowing for synchronization with the program clock associated with the video582and/or audio583streams. In some implementations, the media server552and/or subtitle management circuitry553is configured to superimpose lines of text of the newly-generated subtitle stream (e.g.,588,592) onto respective time-synchronized video frames of the video stream582in a graphical manner. That is, according to some implementations, video frames of the video stream582may be modified to have respective lines of text superimposed thereon, such that the video frames include the newly-generated subtitle text generated on-demand by the media server/controller551. The transport stream585transmitted to the client530may therefore include the media stream580with the modified video stream582having the new subtitles superimposed thereon, wherein the media stream580may be transported with or without a separate subtitle file590.

With the video stream582modified to include superimposed subtitle information thereon, the media player532may not need to play a separate subtitle stream, but rather simply plays the video stream, which presently includes the new on-demand subtitles integrated with the video frames thereof. In some embodiments, the modified video stream may be modified and stored/copied in/to a temporary storage (e.g., volatile memory) of the data storage server552without the native media stream580stored in the non-volatile data storage540being modified. For example, in response to the on-demand subtitle request and/or media content request from the client530, the media server/controller552may retrieve the native media stream580from the non-volatile data storage and copy/buffer the media stream580in temporary storage. With the media stream buffered to/in temporary storage, the subtitle management circuitry553may access the audio stream583, generate new subtitles based thereon using the transcriber circuitry555, and superimpose the new subtitles on the video frames of the temporary copy of the video stream582. For example, the subtitle management circuitry553may access the video stream, generate new video frames representing copies of the original video frames but with the new subtitles superimposed thereon, and overwrite/modify the video stream to include the newly-generated video frames with the subtitles superimposed thereon.

As an alternative to superimposing the new subtitles on the video stream582, the transport stream585may include the original video stream582as part of the media stream580, wherein the new subtitle file590may be played by the media player532with the video and audio streams to produce the on-demand subtitles in the graphical layer of the media presentation.

According to some use cases, the client530may be permitted to configure the language of interest for the requested on-demand subtitles. A known language conversion model of the transcriber555, which is internal to the data storage server550, can insert the new subtitle stream in response to the requested subtitle language rather than the native subtitle stream586. Such embodiments can allow the user to use any media, without language restriction, through in-house storage device options provided by the data storage server550.

FIG.6illustrates a sequence diagram for implementing on-demand subtitle generation and provision in accordance with one or more embodiments of the present disclosure. The process illustrated inFIG.6may allow for on-demand subtitle requests and generation by a data storage server650. The process600ofFIG.6may involve or be implemented using a network-attached data storage device/server, or other type of data storage server. The data storage server650may be configured to provide in-storage subtitle creation in response to on-demand user preferences associated with the client630and/or user.

At operation (0), the process600involves receiving, using user input/output circuitry or other means637, a request to use/access a media file/stream680. For example, the media application635may receive the request. In some implementations, certain subtitle preferences601may be determined by the media application635of the client system630in some manner. For example, in some embodiments, the subtitle preference data601may be received using the user I/O circuitry/means637. The user I/O means637may comprise one or more keyboard, mouse, touchscreen, or audible command devices. In some embodiments, the media application635may access stored/known subtitle preference information relating to a particular user. Generally, the subtitle preference data601may indicate an on-demand subtitle generation request for a version of the requested media content having associated therewith viewable subtitles of a particular language and/or having one or more other desired characteristics. The subtitle preference data601may be provided to the media application635at operation (1).

At operation (2), the process600involves requesting, by the media application635, a media stream680corresponding to the user media request to a data storage server650, such as a network attached storage systems/devices, as described in detail herein. In some implementations, the request from the media application635to the data storage server650may include the desired on-demand subtitle preferences601. In some implementations, the subtitle preference data601may be provided by the media application635as a separate request from the media file/stream request. In some implementations, the request at operation (2) is implemented as a data structure/packet including a region or field configured or utilized to store subtitle preference data.

An interface may be provided/displayed to the user to allow the user to engage with the media application635and/or data storage server650and provide subtitle option selection for the user. For example, the interface may allow for the user to select among a plurality of available languages corresponding to the transcribing functionality of the data storage server650and/or subtitle manager653thereof. The client630may include a storage driver that converts the user input to a specific command.

At operation (3), the media/file server652of the data storage server650accesses the requested media stream680from a nonvolatile data store640of the data storage server650. At operation (4), the media stream680is transferred from the data store640to certain memory, such as volatile memory, of the data storage server650, which may be used to buffer the media stream680for at least a portion of time. That is, the data storage server650(e.g., NAS device/system) performs data retrieval from the data store640internally. In some cases, the media stream680includes multiple audio streams, such as audio streams corresponding to multiple different languages, wherein the media server652is configured to select an audio stream683of the media stream680that the subtitle manager653is equipped to process for text-generation purposes.

At operation (5), the process600involves extracting or otherwise accessing one or more audio streams683of the media stream680. For example, the audio stream(s)683may be provided to a functional subtitle manager module or circuitry653. The subtitle manager circuitry553may be configured to decode the audio stream683, as indicated at operation (6), and generate text subtitles using speech-to-text functional circuitry, as indicated as operation (7). At operation (8), the process600can involve generating, by the subtitle manger653, a new subtitle file690including the newly-generated subtitle text and/or certain timestamp data for use in temporal synchronization of the new subtitle text with the frame(s) of the associated video stream.

In some implementations, as shown as operation (10), the subtitle manager653may be configured to store the new subtitle file690in the nonvolatile data store640. For example, subtitle file690may be stored separately from the native media stream680with which it is associated. Storage of the subtitle file690in a separate flash block from the native media stream680may allow for application of different endurance-related parameters and/or actions to the subtitle file690than is/are applied to the media stream680.

When the newly-generated subtitle file690is stored separately in a separate storage region from the media stream680, the subtitle file690may be appended to the media stream680and played only in response to an on-demand request for subtitle creation/provision from the user/client. Storing the subtitle file690in a separate region, such as a different namespace, from the media stream680can enable the data storage server to phase the subtitle file690relatively easily when it is determined by the data storage server650that the subtitle file690is no longer required. That is, storage of the subtitle file690in a separate region or namespace can improve garbage collection efficiency for the data storage server650with respect to the region, block, or namespace in which the subtitle file690is stored.

The media server652of the data storage server650may be configured to transmit a media transport stream685including the video data of the media stream680, as well as the new subtitle file690, which includes relevant timestamp/temporal information associated with the newly-generated subtitle text. The media application635may be configured to play the video stream of the transport stream685on a display670along with time-synched portions699of the new subtitle text data included in the subtitle file690.

By performing the subtitle generation at the data storage server650, the process600ofFIG.6allows for on-demand subtitle generation without requiring the media stream680to be transferred from the data storage server650to the client630for generation of the subtitles. That is, the data source server650is configured to internally implement the subtitle-generation functionality and provide the packaged data stream685including with the newly-generated subtitles690together with the media stream680.

FIG.7is a flow diagram illustrating a process700for client-side on-demand subtitle request and acquisition in accordance with one or more embodiments of the present disclosure.

At block702, the process700involves receiving a media access request from a user. For example, the media access request may be received using certain user input/output circuitry of the client system. In some embodiments, the client may be configured to present one or more interfaces to the user that allow for selection by the user of certain media items. Such interfaces may further allow for input of on-demand subtitle requests from the user.

The operation(s) associated with block704, which may or may not be implemented as part of the process700, involve receiving a subtitle request or preference from the user using the user input/output circuitry. The determination or receipt of subtitle request/preference information from the user may temporally occur prior to receiving the media access request at block702. For example, in some implementations, the user may create a profile indicating certain subtitle preferences and/or such preferences may be determined/derived by the client from behavioral analysis relating to the user. For example, the client may be configured to determine language usage/preferences of the user based on media selections and/or other behavioral actions of the user vis-à-vis interactions with the client and/or interactions with one or more other systems, wherein information relating to such interactions is provided to or otherwise known by the client.

At block706, the process700involves requesting the selected media stream from a data storage server communicatively coupled to the client. For example, the data storage server may comprise one or more devices or systems, such as a network-attached storage drive. At block708, the process700further involves requesting on-demand subtitles from the data storage device/server, wherein such subtitle request may be incorporated in the request associated with block706. For example, the request associated with block706may involve the transmission of a data packet including one or more fields or data structures indicating subtitle preferences/requests.

At block710, the process700involves receiving from the data storage server the requested media stream along with a new subtitle file/stream generated by the data storage server/device in response to the request(s) associated with block(s)706and/or708. In some implementations, the media stream may include a video stream having the newly-generated subtitles superimposed on video frames thereof rather than, or in addition to, being included in a separate subtitle file. In some implementations, the newly-generated subtitle text is appended to the native media stream associated with the selected/requested media content item.

At block712, the process700involves playing the received media with the new subtitles, wherein temporal presentation of the subtitles may be in accordance with certain timestamp/temporal information incorporated in the new subtitle file received from the data storage server.

FIG.8is a flow diagram illustrating a process800for on-demand subtitle generation and provision by a data storage server/device in accordance with one or more embodiments of the present disclosure.

At block802, the process800involves receiving a media access request from a client device or system, wherein the media access request indicates a desired media content item. In some implementations, the process further involves receiving, either with the request associated with block802or other separate request from the client, certain on-demand subtitle preferences/request from the client. In some implementations, the on-demand subtitle preferences of the user/client may be determined by the data storage server independent of a present request therefor. For example, the data storage device/server may be configured to determine subtitle preferences of the user/client based on user profile data maintained by the data storage server/device. Such subtitle preference profile data may be received from or set by the client or otherwise obtained/determined by the data storage device/server.

At block806, the process800involves decoding one or more audio streams associated with the requested media content item, which may be retrieved from a nonvolatile data store of the data storage device/server. That is, the data storage server/device may decode the audio corresponding to the subtitle request.

At block808, the process800involves transcribing the audio stream(s) to generate subtitles in accordance with the on-demand subtitle preferences of the client/user. The data storage device/server can perform speech-to-text conversion according to language preferences specified by the request from the client.

At block810, the process800involves generating a new subtitle file based on the transcribed subtitles associated with the operation at block808. For example, the operation of block808may produce certain subtitle text data, which may be incorporated in the new subtitle file. Furthermore, the new subtitle file may have appended thereto and/or otherwise incorporated therewith certain timestamp or other temporal data associating respective lines of text of the new subtitle text with presentation times relating to video frame times of the associated video stream.

At block812, the process800involves transmitting the native media stream, including the relevant video and audio streams thereof, as well as the new subtitle file, to the client. The process800may further comprise, at block814, storing the subtitle file including relevant timestamps. In some embodiments, the subtitle file is stored in a different endurance group of the nonvolatile data storage relative to the endurance group in which the native media stream associated with the newly-generated subtitle file is stored.

Additional Embodiments

Those skilled in the art will appreciate that in some embodiments, other types of video file seeking and/or live transcoding systems can be implemented while remaining within the scope of the present disclosure. In addition, the actual steps taken in the processes discussed herein may differ from those described or shown in the Figures. Depending on the embodiment, certain of the steps described above may be removed, others may be added.

While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of protection. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the protection. For example, the various components illustrated in the Figures may be implemented as software and/or firmware on a processor, ASIC/FPGA, or dedicated hardware. Also, the features and attributes of the specific embodiments disclosed above may be combined in different ways to form additional embodiments, all of which fall within the scope of the present disclosure. Although the present disclosure provides certain preferred embodiments and applications, other embodiments that are apparent to those of ordinary skill in the art, including embodiments which do not provide all of the features and advantages set forth herein, are also within the scope of this disclosure. Accordingly, the scope of the present disclosure is intended to be defined only by reference to the appended claims.

All of the processes described above may be embodied in, and fully automated via, software code modules executed by one or more general purpose or special purpose computers or processors. The code modules may be stored on any type of computer-readable medium or other computer storage device or collection of storage devices. Some or all of the methods may alternatively be embodied in specialized computer hardware.