Patent Description:
In a streaming audio asset there are often times constraints on processing of the audio asset such that a large buffer of audio information cannot be maintained. These constraints may either be due to physical limitations of the system or constraints within the program streaming the audio asset.

One such situation where there are constraints on the amount of information that can be buffered is in the context of cloud based emulation. In a cloud-based emulation system the majority of the processing takes place on the remote cloud-based server. This allows the client device platform that is communicating with the cloud-based server to use less resources for processing the cloud based game. Current techniques implement incremental buffering to make the game load faster and parallel processing between the server and the client. However as a result of this arrangement communication between the client device and the server must be synchronized and the client device can only maintain a small buffer to of emulated information. If the client device and the server are not synchronized it may result in unusual or unexpected device behavior. Sometimes audio blocks that are sent through the network are lost or the system requires additional time to process the audio which results in a loss of synchronization. Thus there is a need in the art for a way to accelerate processing audio to ensure synchronization and improve loading times for audio assets. Previously proposed arrangements are disclosed in <CIT>, <CIT> and T.

The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:.

Depending on the frequency of the sound the human ear can distinguish meaningful short changes in frequency as fast as <NUM>. Though research in this field is ongoing some limits have been found empirically. Detection of meaningful changes in frequency by the human ear varies greatly depending on the frequency of the waveform in question. For example an audible low frequency bass signal at <NUM> (Hertz) Hz measures <NUM> milliseconds (ms) crest to crest. It would take more than <NUM> to produce a meaningful sound that would be recognized as something other than noise. Conversely a high frequency waveform of <NUM> kilohertz (kHz) becomes recognizable sound much sooner, at around <NUM>. When streaming audio a continuous soundwave is parsed in to samples which are packaged in to discrete packets or audio blocks and sent over the network to a client. The client device receives the audio block and processes it to place the audio block in a form which is readily playable on the client device. In prior art systems the client device may include a large buffer of audio blocks which are processed before playback. In this way prior art systems were able to maintain synchronization of playback and processing because playback is generally slower than processing and any time processing slowed down there was a large buffer before playback needs to stop and wait for processing to finish. There are situations where systems cannot maintain a large buffer for example in systems with small amounts of physical memory or in systems that must maintain synchronization with a server; these limitations are referred to herein as real time constraints. In these instances audio processing may be accelerated if the system can select and drop certain audio blocks before they are processed without a user noticing the effects of the dropped audio block.

Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention, which is defined by the appended claims. Accordingly, various implementations of aspects of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.

<FIG> shows a method <NUM> for selectively dropping and blending audio blocks for acceleration of an audio stream. Initially a device may receive or generate an audio stream, as indicated at <NUM>. The audio stream may be composed of a series of packets or audio blocks containing audio information. The packets or audio block size may be predefined by the networking protocol or underlying standard for convenience and to provide minimal latency impact. As an added benefit these packet or block sizes are usually well aligned with the minimum audio latency response of the underlying systems. In other embodiments the packet or audio block size may be defined by the system to maximize the perceived audio quality when dropping blocks according to aspects of the present disclosure. It should be noted that using audio block sizes that are not a factorial of the block size of the system inherently increases the latency of processing. Additionally the audio stream may be subject to real time constraints that limit the amount of buffered information that may be kept for the stream or the amount of time before the audio is played back.

The device may determine a location in the audio stream to drop an audio block, as indicated at <NUM> such as an audio block that has no dependencies or a block for which there is or is not a large audio level difference between the end of previous block and the beginning of the next block. In one embodiment the location to drop an audio block is determined from signal analysis which may better be understood with respect to <FIG>.

The location to drop a block may be determined by transforming the waveform <NUM> into frequency and time domains using known methods such as Fourier transform (DFT), discrete cosine transform (DCT), or discrete wavelet transform (DWT) analysis. In analysis of the transform, the wave spectra of each audio block is chopped <NUM> in to wavelets and distinct the change between wavelets is measured <NUM>, blocks that exhibit a high degree of changes over time are flagged and may not be dropped <NUM>.

A high degree of rate of change may be determined by a threshold. The device may be configured to drop audio blocks at a set interval to accelerate audio processing hereinafter referred to as a drop interval. In an embodiment if the block determined to be dropped by the block interval is flagged as having a high rate of change the system will choose the next block in the stream, in a different embodiment the system may choose the previous block in the stream. The drop interval may be determined by the system based on latency constraints for example when the system determines high latency in processing blocks, the drop interval may be reduced. Conversely the system may increase the drop interval when it is determined that the latency is low. In an alternative embodiment the device may have a predetermined threshold for audio levels and when the audio level is below that threshold the device may drop blocks. The device may also have some combination of audio level threshold and drop interval.

<FIG> shows a method for determining blocks to be dropped, not forming part of the invention.

The method may begin with determining the drop interval <NUM> from latency information as discussed above. Signal analysis as discussed above may be used to transform the waveform into frequency time domain waves <NUM>, break the waves into wavelets <NUM> and analyze the rate of change of the wavelets <NUM>. From the rate of change information a drop priority flag may be applied to each audio block <NUM> which indicates the amount of changes over time. Blocks with a high degree of change will be flagged as low priority blocks whereas blocks with a low degree of change will be flagged with a higher priority. The system may use the drop interval to set a sliding priority window <NUM>. The system applies the drop priority flags to each of the blocks within the sliding priority window <NUM>. Within the sliding priority window the system determines which blocks drop <NUM> by comparing the drop priority flag values of each block. The block with the highest priority <NUM>(i.e. the block with the lowest amount of changes over time) will be dropped and the other lower priority blocks will be kept <NUM>. The size of the priority window can be of any length but it is preferable that the perceived latency of the system is reduced. Ideally the priority window is under <NUM>. In this case, the drop interval may set the length of time between each priority window.

Aspects of the method of <FIG> may better be under with respect to <FIG>. A waveform <NUM> of streaming audio content is depicted. Line <NUM> represents the amount of streaming audio that has been stored in memory. The <NUM> and <NUM> represent the sliding priority frame. The sliding priority frames occur every drop interval <NUM>. Within the sliding priority frame the block with the highest drop priority is dropped <NUM>.

Once the device has determined that dropping an audio block is appropriate, the device may start dropping audio blocks. To drop an audio block the device may bypass processing of the block, as indicated at <NUM>. Alternatively the device may initially receive the unprocessed audio block in a buffer and allow the unprocessed audio block to be over-written by the next audio block. The device may then receive the next audio block after the dropped block, as indicated at <NUM> and process it for playback. A key concept of the present disclosure is that the described system does not add any additional latency to the processing of audio received over the network. Once an audio packet has been received from the network and decompressed, the client can analyze and cut from the stream without incurring any additional latency.

The device compensates for the dropped audio block <NUM> by reducing the difference in audio level between the end of the prior audio block and the beginning of the next audio block. According to the invention, the device averages the last set samples of the prior audio block with initial set sample of the next audio block and adjust the values of those samples towards that average. The number of samples used for the average may be determined empirically to find the most natural sounding way to bridge a dropped block. For example it has been found that pops and clicks are effectively removed by averaging audio of only <NUM> in length. According to the invention, wave form analysis is performed to determine the length of audio levels to modify as seen in <FIG>. The frequency of the previous audio block or blocks is analyzed <NUM>. If the analysis determines <NUM> that the wave form is low frequency <NUM>, a longer period of averaging is chosen <NUM> for a better match to the original spectra. Conversely if the waveform is determined <NUM> to be high frequency <NUM> then a very short period of averaging or no averaging will be chosen <NUM>. By way of example and not by way of limitation for a low frequency waveform the system may choose an averaging period equal to or less than <NUM> and for high frequency waveforms the system may choose an averaging period equal to or less than <NUM>. According to alternate aspects of the present disclosure there may be greater than two averaging periods, for example there may be averaging periods for low, middle and high frequency waveforms or there may be a number of averaging periods corresponding to wave form frequencies chosen to best approximate the underlying waveforms without noticeably interfering with the user experience. In some implementation the gap in audio level created by the dropped block may be filled by interpolating the waveform using wavelet analysis. In some embodiments the length of interpolation is chosen based on the frequency of the wave form. As discussed above a longer period of the interpolation calculation will be chosen for a lower frequency waveform and short period of interpolation will be chosen for a high frequency audio wave form.

Once the device has dropped a block and compensated for the dropped block as discussed above it may play the audio, as indicated at <NUM> and/or return to receiving and/or generating audio blocks as normal, as indicated at <NUM>.

<FIG> shows graph of audio level versus time depicting <NUM> milliseconds of a sound wave <NUM>.

In this depiction the wave is shown as a continuous line but it should be understood that in certain embodiments the wave may be composed of a multiple samples of audio information. For example a typical Compact Disk quality MP3 audio file is composed of 44100samples per second. <FIG> depicts a soundwave <NUM> with a dropped block <NUM> according to aspects of the present disclosure. In the illustrated example, the dropped block <NUM> lasts for <NUM> milliseconds but in other embodiments the block may be as short as <NUM> milliseconds or as long as <NUM> milliseconds. In this depiction the audio is streaming as represented by the dotted line <NUM>. The device already decided to drop a block <NUM> and has received the next audio block <NUM> after the dropped block <NUM>. The devices has not yet compensated for the dropped block <NUM>.

It should be understood that there is no minimum length for dropped blocks but as the number of dropped blocks increase and the length of the dropped blocks shortens the human ear begins to perceive the sound at a higher frequency. For example if the dropped block length was <NUM> and every other block was dropped the perceived audio would sound accelerated and at a very high pitch. Currently the optimal block length is between <NUM> and <NUM>. In current multimedia systems block lengths of greater than <NUM> are not found because <NUM> corresponds to the current industry standard for video of <NUM> frames per seconds.

<FIG> depicts a soundwave <NUM> with a dropped block <NUM> and compensation according to aspects of the present disclosure. In the disclosed embodiment the tail end of the previous block <NUM> and the beginning of the next block <NUM> have been averaged to eliminate user perception of the dropped block. By averaging the beginning and end of the blocks the user's perception of clicks and pops may be reduced or eliminated completely. The dashed line represents the original audio <NUM> and is included to show the difference between the original audio <NUM> and the dropped block-compensated audio <NUM>.

<FIG> shows a soundwave <NUM> with two dropped blocks <NUM>, <NUM> and compensation according to aspects of the present disclosure. In this embodiment the device has streamed <NUM> of audio <NUM>. The device is configured with a drop interval <NUM> by way of example and not by way of limitation the device may be configured to drop blocks at a <NUM> interval. The interval may be determined experimentally to find the optimal number of dropped blocks to provide desired level of audio acceleration without a perceived audio quality loss. The device has identified and dropped a first audio block <NUM> and averaged the end of the previous audio block <NUM> with the beginning of the next audio block <NUM> and averaged the beginning of the next audio block <NUM> with the previous audio block <NUM> to compensate for the dropped audio. The device has then continued streaming audio till the end of the interval <NUM>. At which point the device drops another audio block <NUM>. After receiving the beginning of the next audio block <NUM> the device averages the end of the previous un-dropped audio block <NUM> by the beginning of the next audio block <NUM> and the beginning of the next audio block <NUM> with the end of the previous audio block <NUM> to compensate for the dropped block. At which point the device is able to play back the compensated audio.

The block diagram shown in <FIG> schematically illustrates certain aspects of the present disclosure within the particular context of remote emulation. In this example, an emulator <NUM> may be accessed by a device <NUM> over a network <NUM>. The device <NUM> may access alternative remote emulators <NUM> over the network <NUM> (herein referred to as remote emulators). Emulators <NUM> may be identical to each other, or they may each be programed to emulate unique legacy programs <NUM> including unique legacy games <NUM>.

The device <NUM> may include a central processor unit (CPU) <NUM>. By way of example, a CPU <NUM> may include one or more processors, which may be configured according to, e.g., a dual-core, quad-core, multi-core, or Cell processor architecture. The device <NUM> may also include a memory <NUM> (e.g., RAM, DRAM, ROM, and the like). The CPU <NUM> may execute a process-control program <NUM>, portions of which may be stored in the memory <NUM>. The device <NUM> may also include well-known support circuits <NUM>, such as input/output (I/O) circuits <NUM>, power supplies (P/S) <NUM>, a clock (CLK) <NUM> and cache <NUM>. The device <NUM> may optionally include a mass storage device <NUM> such as a disk drive, CD-ROM drive, tape drive, or the like to store programs and/or data. The device <NUM> may also optionally include a display unit <NUM> and a user interface unit <NUM> to facilitate interaction between the device <NUM> and a user who requires direct access to the device <NUM>. The display unit <NUM> may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numerals, or graphical symbols. The user interface unit <NUM> may include a keyboard, mouse, joystick, light pen, or other input devices. Additionally the device may include an audio device <NUM> for the playback or recording of audio waveforms. Such audio devices may include speakers, microphones, oscilloscopes, phonographs and other sound playback/recording media. The device <NUM> may control the audio device through appropriate driver software, which may be stored in the memory <NUM> and/or storage device <NUM> and executed by the CPU <NUM>. The device <NUM> may include a network interface <NUM>, configured to enable the use of Wi-Fi, an Ethernet port, or other communication methods.

The network interface <NUM> may incorporate suitable hardware, software, firmware or some combination of two or more of these to facilitate communication via an electronic communications network <NUM>. The network interface <NUM> may be configured to implement wired or wireless communication over local area networks and wide area networks such as the Internet. The device <NUM> may send and receive data and/or requests for files via one or more data packets over the network <NUM>.

The preceding components may exchange signals with each other via an internal system bus <NUM>. The device <NUM> may be a general purpose computer that becomes a special purpose computer when running code that implements embodiments of the present invention as described herein.

The emulator <NUM> may include a central processor unit (CPU) <NUM>'. By way of example, a CPU <NUM>' may include one or more processors, which may be configured according to, e.g., a dual-core, quad-core, multi-core, or Cell processor architecture. The emulator <NUM> may also include a memory <NUM>' (e.g., RAM, DRAM, ROM, and the like). The CPU <NUM>' may execute a process-control program <NUM>', portions of which may be stored in the memory <NUM>'. The emulator <NUM> may also include well-known support circuits <NUM>', such as input/output (I/O) circuits <NUM>', power supplies (P/S) <NUM>', a clock (CLK) <NUM>' and cache <NUM>'. The emulator <NUM> may optionally include a mass storage device <NUM>' such as a disk drive, CD-ROM drive, tape drive, or the like to store programs and/or data. The emulator <NUM> may also optionally include a display device <NUM>' and user interface unit <NUM>' to facilitate interaction between the emulator <NUM> and a user who requires direct access to the emulator <NUM>. By way of example and not by way of limitation a device or engineer <NUM> may need direct access to the emulator <NUM> in order to program the emulator <NUM> to properly emulate a desired legacy program <NUM> or to add additional capabilities to a legacy program <NUM>. The display device <NUM>' may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numerals, or graphical symbols. The user interface unit <NUM>' may include a keyboard, mouse, joystick, light pen, or other device. The emulator <NUM> may include a network interface <NUM>', configured to enable the use of Wi-Fi, an Ethernet port, or other communication methods.

The network interface <NUM>' may incorporate suitable hardware, software, firmware or some combination of two or more of these to facilitate communication via the electronic communications network <NUM>. The network interface <NUM>' may be configured to implement wired or wireless communication over local area networks and wide area networks such as the Internet. The emulator <NUM> may send and receive data and/or requests for files via one or more data packets over the network <NUM>.

The emulator <NUM> may be a general purpose computer that becomes a special purpose computer when running code that implements embodiments of the present invention as described herein. Emulator <NUM> may access a legacy program <NUM> that has been selected by the device <NUM> for emulation through the internal system bus <NUM>'. There may be more than one legacy program <NUM> stored in the emulator. The legacy programs may also be stored in the memory <NUM>' or in the mass storage device <NUM>'. Additionally, one or more legacy programs <NUM> may be stored at a remote location accessible to the emulator <NUM> over the network <NUM>. Each legacy game <NUM> contains game code and legacy audio<NUM>.

The game code <NUM> may contain un-accelerated audio that is sent from the emulator <NUM> to the device <NUM>. The device <NUM> may have real time constraints and implement the disclosed method for accelerating audio <NUM>.

While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. Therefore, the scope of the present invention should be determined not with reference to the above description but should, instead, be determined with reference to the appended claims.

Claim 1:
A method for accelerated audio processing in a streaming environment the method comprising:
a) determining, from signal analysis for a series of audio blocks in a set interval for a streaming audio asset, an audio block to be dropped from the streaming audio asset for the set interval;
b) dropping the determined audio block ;
c) compensating for the dropped audio block to generate compensated audio; and
d) playing the compensated audio through an audio device,
characterised in that compensating at c) comprises averaging a last set of samples of a prior audio block that is immediately before the dropped audio block with an initial set of samples of a next audio block, and adjusting the values of those samples towards that average, in which
frequency analysis is applied to the prior audio block immediately before the dropped audio block to determine a period, of the end of the prior audio block to be averaged with the beginning of the next audio block, so that a greater amount of the end of the prior audio block is averaged for a lower frequency waveform than for a higher frequency waveform.