Patent ID: 12190013

DETAILED DESCRIPTION

This disclosure relates generally to computer ecosystems including aspects of consumer electronics (CE) device networks such as but not limited to computer game networks. A system herein may include server and client components which may be connected over a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices including game consoles such as Sony PlayStation® or a game console made by Microsoft or Nintendo or other manufacturer, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g. smart TVs, Internet-enabled TVs), portable computers such as laptops and tablet computers, and other mobile devices including smart phones and additional examples discussed below. These client devices may operate with a variety of operating environments. For example, some of the client computers may employ, as examples, Linux operating systems, operating systems from Microsoft, or a Unix operating system, or operating systems produced by Apple, Inc., or Google. These operating environments may be used to execute one or more browsing programs, such as a browser made by Microsoft or Google or Mozilla or other browser program that can access websites hosted by the Internet servers discussed below. Also, an operating environment according to present principles may be used to execute one or more computer game programs.

Servers and/or gateways may include one or more processors executing instructions that configure the servers to receive and transmit data over a network such as the Internet. Or, a client and server can be connected over a local intranet or a virtual private network. A server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, etc.

Information may be exchanged over a network between the clients and servers. To this end and for security, servers and/or clients can include firewalls, load balancers, temporary storages, and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus that implement methods of providing a secure community such as an online social website to network members.

A processor may be a single- or multi-chip processor that can execute logic by means of various lines such as address lines, data lines, and control lines and registers and shift registers.

Components included in one embodiment can be used in other embodiments in any appropriate combination. For example, any of the various components described herein and/or depicted in the Figures may be combined, interchanged, or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and/or A, B, and C together, etc.

Now specifically referring toFIG.1, an example system10is shown, which may include one or more of the example devices mentioned above and described further below in accordance with present principles. The first of the example devices included in the system10is a consumer electronics (CE) device such as an audio video device (AVD)12such as but not limited to an Internet-enabled TV with a TV tuner (equivalently, set top box controlling a TV). The AVD12alternatively may also be a computerized Internet enabled (“smart”) telephone, a tablet computer, a notebook computer, a HMD, a wearable computerized device, a computerized Internet-enabled music player, computerized Internet-enabled head phones, a computerized Internet-enabled implantable device such as an implantable skin device, etc. Regardless, it is to be understood that the AVD12is configured to undertake present principles (e.g., communicate with other CE devices to undertake present principles, execute the logic described herein, and perform any other functions and/or operations described herein).

Accordingly, to undertake such principles the AVD12can be established by some or all of the components shown inFIG.1. For example, the AVD12can include one or more displays14that may be implemented by a high definition or ultra-high definition “4K” or higher flat screen and that may be touch-enabled for receiving user input signals via touches on the display. The AVD12may include one or more speakers16for outputting audio in accordance with present principles, and at least one additional input device18such as an audio receiver/microphone for entering audible commands to the AVD12to control the AVD12. The example AVD12may also include one or more network interfaces20for communication over at least one network22such as the Internet, an WAN, an LAN, etc. under control of one or more processors24. A graphics processor24A may also be included. Thus, the interface20may be, without limitation, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as but not limited to a mesh network transceiver. It is to be understood that the processor24controls the AVD12to undertake present principles, including the other elements of the AVD12described herein such as controlling the display14to present images thereon and receiving input therefrom. Furthermore, note the network interface20may be a wired or wireless modem or router, or other appropriate interface such as a wireless telephony transceiver, or Wi-Fi transceiver as mentioned above, etc.

In addition to the foregoing, the AVD12may also include one or more input ports26such as a high definition multimedia interface (HDMI) port or a USB port to physically connect to another CE device and/or a headphone port to connect headphones to the AVD12for presentation of audio from the AVD12to a user through the headphones. For example, the input port26may be connected via wire or wirelessly to a cable or satellite source26aof audio video content. Thus, the source26amay be a separate or integrated set top box, or a satellite receiver. Or, the source26amay be a game console or disk player containing content. The source26awhen implemented as a game console may include some or all of the components described below in relation to the CE device44.

The AVD12may further include one or more computer memories28such as disk-based or solid state storage that are not transitory signals, in some cases embodied in the chassis of the AVD as standalone devices or as a personal video recording device (PVR) or video disk player either internal or external to the chassis of the AVD for playing back AV programs or as removable memory media. Also in some embodiments, the AVD12can include a position or location receiver such as but not limited to a cellphone receiver, GPS receiver and/or altimeter30that is configured to receive geographic position information from a satellite or cellphone base station and provide the information to the processor24and/or determine an altitude at which the AVD12is disposed in conjunction with the processor24. The component30may also be implemented by an inertial measurement unit (IMU) that typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the location and orientation of the AVD12in three dimensions.

Continuing the description of the AVD12, in some embodiments the AVD12may include one or more cameras32that may be a thermal imaging camera, a digital camera such as a webcam, and/or a camera integrated into the AVD12and controllable by the processor24to gather pictures/images and/or video in accordance with present principles. Also included on the AVD12may be a Bluetooth transceiver34and other Near Field Communication (NFC) element36for communication with other devices using Bluetooth and/or NFC technology, respectively. An example NFC element can be a radio frequency identification (RFID) element.

Further still, the AVD12may include one or more auxiliary sensors37(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer, or a magnetic sensor, an infrared (IR) sensor, an optical sensor, a speed and/or cadence sensor, a gesture sensor (e.g. for sensing gesture command), etc.) providing input to the processor24. The AVD12may include an over-the-air TV broadcast port38for receiving OTA TV broadcasts providing input to the processor24. In addition to the foregoing, it is noted that the AVD12may also include an infrared (IR) transmitter and/or IR receiver and/or IR transceiver42such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD12, as may be a kinetic energy harvester that may turn kinetic energy into power to charge the battery and/or power the AVD12.

Still referring toFIG.1, in addition to the AVD12, the system10may include one or more other CE device types. In one example, a first CE device44may be a computer game console that can be used to send computer game audio and video to the AVD12via commands sent directly to the AVD12and/or through the below-described server while a second CE device46may include similar components as the first CE device44. In the example shown, the second CE device46may be configured as a computer game controller manipulated by a player or a head-mounted display (HMD) worn by a player47. In the example shown, only two CE devices44,46are shown, it being understood that fewer or greater devices may be used. A device herein may implement some or all of the components shown for the AVD12. Any of the components shown in the following figures may incorporate some or all of the components shown in the case of the AVD12.

Now in reference to the afore-mentioned at least one server50, it includes at least one server processor52, at least one tangible computer readable storage medium54such as disk-based or solid state storage, and at least one network interface56that, under control of the server processor52, allows for communication with the other devices ofFIG.1over the network22, and indeed may facilitate communication between servers and client devices in accordance with present principles. Note that the network interface56may be, e.g., a wired or wireless modem or router, Wi-Fi transceiver, or other appropriate interface such as, e.g., a wireless telephony transceiver.

Accordingly, in some embodiments the server50may be an Internet server or an entire server “farm”, and may include and perform “cloud” functions such that the devices of the system10may access a “cloud” environment via the server50in example embodiments for, e.g., network gaming applications. Or, the server50may be implemented by one or more game consoles or other computers in the same room as the other devices shown inFIG.1or nearby.

FIG.2illustrates that a sound effects (SFX) supplier computer200may supply SFX multitrack containers202to a customer computer204, such as a computer system of a game developer, typically over wired and/or wireless paths of a wide area computer network such as the Internet.

A multitrack container202includes both a multitrack (two and typically more than two tracks) SFX audio file in, e.g., WAV format or MP3 format, and track modification information pertaining to the tracks. As discussed further below, the track modification information is useful to modify the SFX file. That is, a game developer operating the customer computer204can, using the track modification information, alter the SFX file for rendering in a computer application such as a computer simulation such as a computer game developed by the first developer. Likewise, a different developer can receive the same multitrack container202and use the track modification information to alter same SFX file in a different way for a different computer game. As discussed further below, the track modification information can be used to alter the SFX file at run time of the computer game or offline from run time.

FIG.3illustrates example overall logic summarized above. Commencing at block300, SFX tracks can be imported into a digital audio workstation (DAW) such as may be part of the supplier computer200shown inFIG.2. Moving to block302, using the DAW, SFX files are created with each having plural audio tracks. Track modification information, which may be referred to herein as “metadata”, is associated with respective SFX files to produce a multitrack container.

Moving to block304, the multitrack containers are provided to customers, who may modify the SFX files within the containers using the accompanying track modification information. If desired, an SFX rendering engine may also be provided to the customer computers at block306. The rendering engine may include an authoring tool. An example of such an engine is Wave Works Interactive Sound Engine (Wwise) from Audiokinetic. Using the authoring tool executed on the customer computer200, at block308the customer game developer makes and manages changes to the SFX files as desired using the respective track modification information accompanying the SFX files.

Turn now toFIG.4, which illustrates a screen shot that may be provided on a display400such as any display described herein. The screen shot illustrates track modification information associated with a SFX file such as a WAV or MP3 file. As shown, the track modification information illustrates plural tracks that make up the SFX file, in the example shown, a bass track402, an AC GTR track404, an EG track406, a PAD01 track408, a Lead Vox Eng track410, and a Back End track412. Respective sound images414may be presented for each track, representing the particular contribution of that track to the overall SFX file.

Also, the screen shot illustrates that for each track, parameters416of the track can be presented. The parameters include, without limitation, volume, delay, attenuation, compression, distortion, and reverberation, panning, filter, time stretch, pitch shift. Collectively, all of the above information can be regarded as the track modification information that accompanies the SFX multitrack file, and an application developer such as a game developer can modify any such information. Thus, the game developer can modify the parameters of individual tracks and, hence, modify the sound contribution to the SFX of individual tracks, and can remove individual tracks as desired, and make other modifications to the SFX file established by the tracks.

This is illustrated further inFIG.5, which shows a screen shot500that may be presented on, e.g., the customer application developer computer204shown inFIG.2or other display disclosed herein, in the example shown, using Wwise. For simplicityFIG.5illustrates four tracks502for editing. Editing selectors504are provided for each respective track to enable modification of individual tracks or elimination of certain tracks from the final SFX file, an example audio diagram of which is shown in the screen shot600ofFIG.6as indicated at602.FIG.6also indicates that SFX output parameters604may be presented, including, without limitation, channel configuration, make up gain, trim start, trim end, override WAV loop points and loop start and end times, crossfade duration and crossfade shape, marker input mode, marker detection sensitivity, fade-in duration, fade-in curve, and fade-out duration. Note that is non-limiting examples these output parameters604may also be part of the track modification information for each track in the multitrack container202.

Using any of the information discussed herein, a game developer can modify a multitrack SFX to change the delay between tracks, the amplitude of any given track, the attenuation of any give track, and filtering for any track, such as by adding or removing high frequency components or low frequency components of the track. One track may be faded in or out or removed altogether. Compression, reverberation, panning, filter, time stretch, pitch shift, and distortion may be added to one or more tracks. Surround sound may be accommodated by designating some tracks for play on front speakers, other tracks for play on rear speakers, etc.

FIG.7illustrates a simplified SFX700with four tracks702that are rendered into a single SFX sound704. As an example, suppose the SFX700is for a footfall (shoe step). One track702may be for a shoe-pavement scraping sound, another track may be for a heel sound striking pavement, another track may be the toe striking the pavement, and another track may be for the swoosh of clothing as the character walks.

FIG.8illustrates that some tracks800can be combined to render a first layer802of a sound effect while other tracks804can be combined to render a second layer806of the SFX. Rendering may be done offline in which game designers can modify/mix the content to their liking prior to rendering to a single audio file that will then be imported into the sound engine (e.g. Wwise).

Or, as illustrated inFIG.8rendering may be done at game playback runtime according to game calls. In essence, sound designers have the ability to import the multitrack session into the rendering engine (such as Wwise) and then regroup some tracks together on which they attach Realtime parameter control (RTPC) and states that the game engine triggers at runtime. RTPC and States are sound designer-friendly abstractions of game engine application programming interface (API) calls.

FIG.9illustrates visual representations of SFX all derived from a common multitrack container to indicate that many different modifications may be made to the same SFX by different game designers.

FIG.10illustrates runtime logic discussed above. Commencing at block1000, a multitrack container may be received at runtime. Game calls driven by the game situation re received at block1002. Moving to block1004, the called SFX is accessed. Proceeding to block1006, using the track modification information, the SFX file in the container is modified and the modified SFX is rendered at block1008.

It will be appreciated that whilst present principals have been described with reference to some example embodiments, these are not intended to be limiting, and that various alternative arrangements may be used to implement the subject matter claimed herein.