SYSTEMS AND METHODS FOR REAL-TIME QUALITY ANALYSIS OF WIRELESS LIVE AUDIO STREAMS

A method for real-time detection of millisecond gaps in live audio streams includes receiving a data representation of a live audio signal corresponding to a live event via a wireless network and processing the data representation of the live audio signal corresponding to the live event into a live audio stream. The method also includes transmitting, by a first mobile computing device, the live audio stream to a second mobile computing device communicatively coupled to the first mobile computing device. The method also includes detecting a first gap in the received live audio stream and calculating at least one of a timestamp corresponding to the first gap in the received live audio stream or a duration of the first gap in the received live audio stream.

FIELD OF THE INVENTION

This invention relates generally to the field of real-time delivery of data, such as audio, over wireless networks. More specifically, the invention relates to systems and methods for real-time quality analysis of live audio streams delivered over wireless networks.

BACKGROUND

Attendees of live events often bring and use their mobile computing device to stream data (e.g., audio or video) using at least one of the available wireless networks at the venue (e.g., Wi-Fi or cellular). Measuring and analyzing the quality of the streamed data is crucial in order to make any necessary adjustments to improve performance. For example, audio streams may suffer from buffer underrun events which produce gaps in audio. However, these types of audio gaps may be very small (e.g., a few milliseconds) and hard to detect using conventional systems and methods. Therefore, there is a need for systems and methods that can detect and measure audio gaps in a live audio stream.

SUMMARY

The present invention includes systems and methods for real-time detection of millisecond gaps in live audio streams. For example, the present invention includes systems and methods for receiving a data representation of a live audio signal corresponding to a live event via a wireless network and processing the data representation of the live audio signal corresponding to the live event into a live audio stream. The present invention also includes systems and methods for transmitting the live audio stream from a first mobile computing device to a second mobile computing device communicatively coupled to the first mobile computing device. The present invention also includes systems and methods for detecting a gap in the received live audio stream and calculating at least one of a timestamp corresponding to the gap in the received live audio stream or a duration of the gap in the received live audio stream.

In one aspect, the invention includes a computerized method for real-time detection of millisecond gaps in live audio streams. The computerized method includes receiving, by a first mobile computing device, a data representation of a live audio signal corresponding to a live event via a wireless network. The computerized method also includes processing, by the first mobile computing device, the data representation of the live audio signal corresponding to the live event into a live audio stream. The computerized method also includes transmitting, by the first mobile computing device, the live audio stream to a second mobile computing device communicatively coupled to the first mobile computing device.

The computerized method also includes detecting, by the second mobile computing device, a first gap in the received live audio stream. The computerized method also includes calculating, by the second mobile computing device, at least one of a timestamp corresponding to the first gap in the received live audio stream or a duration of the first gap in the received live audio stream.

In some embodiments, the duration of the first gap in the received live audio stream is less than 20 milliseconds. For example, in some embodiments, the duration of the first gap in the received live audio stream is less than 10 milliseconds. In some embodiments, the duration of the first gap in the received live audio stream is less than 5 milliseconds.

In some embodiments, the computerized method further includes detecting, by the second mobile computing device, a second gap in the received live audio stream. For example, in some embodiments, the computerized method further includes calculating, by the second mobile computing device, at least one of a timestamp corresponding to the second gap in the received live audio stream or a duration of the second gap in the received live audio stream. In some embodiments, the computerized method further includes calculating, by the second mobile computing device, a rate of gaps corresponding to the received live audio stream.

In some embodiments, the first mobile computing device and the second mobile computing device are communicatively coupled via a wired connection. In other embodiments, the first mobile computing device and the second mobile computing device are communicatively coupled via a wireless connection.

In some embodiments, the computerized method further includes receiving, by the first mobile computing device, the data representation of the live audio signal corresponding to the live event from an audio server computing device via the wireless network.

In another aspect, the invention includes a system for real-time detection of millisecond gaps in live audio streams. The system includes a first mobile computing device communicatively coupled to a second mobile computing device. The first mobile computing device is configured to receive a data representation of a live audio signal corresponding to a live event via a wireless network. The first mobile computing device is also configured to process the data representation of the live audio signal corresponding to the live event into a live audio stream. The first mobile computing device is also configured to transmit the live audio stream to the second mobile computing device communicatively coupled to the first mobile computing device.

The second mobile computing device is configured to detect a first gap in the received live audio stream. The second mobile computing device is also configured to calculate at least one of a timestamp corresponding to the first gap in the received live audio stream or a duration of the first gap in the received live audio stream.

In some embodiments, the duration of the first gap in the received live audio stream is less than 20 milliseconds. For example, in some embodiments, the duration of the first gap in the received live audio stream is less than 10 milliseconds. In some embodiments, the duration of the first gap in the received live audio stream is less than 5 milliseconds.

In some embodiments, the second mobile computing device is also configured to detect a second gap in the received live audio stream. For example, in some embodiments, the second mobile computing device is also configured to calculate at least one of a timestamp corresponding to the second gap in the received live audio stream or a duration of the second gap in the received live audio stream. In some embodiments, the second mobile computing device is also configured to calculate a rate of gaps corresponding to the received live audio stream.

In some embodiments, the first mobile computing device and the second mobile computing device are communicatively coupled via a wired connection. In other embodiments, the first mobile computing device and the second mobile computing device are communicatively coupled via a wireless connection.

In some embodiments, the system includes an audio server computing device communicatively coupled to the first mobile computing device, the first mobile computing device configured to receive the data representation of the live audio signal corresponding to the live event from the audio server computing device via the wireless network.

These and other aspects of the invention will be more readily understood from the following descriptions of the invention, when taken in conjunction with the accompanying drawings and claims.

DETAILED DESCRIPTION

FIG.1is a schematic diagram of a system architecture100for real-time delivery of live event data over wireless networks, according to an illustrative embodiment of the invention. System100includes one or more mobile computing devices102communicatively coupled to a server computing device104(or audio server computing device) over one or more wireless networks106. Mobile computing device102includes an application110, one or more speakers112, one or more displays114, and one or more microphones116. In some embodiments, the server computing device104is communicatively coupled to an audio interface (not shown).

Exemplary mobile computing devices102include, but are not limited to, tablets and smartphones, such as Apple® iPhone®, iPad® and other iOS®-based devices, and Samsung® Galaxy®, Galaxy Tab™ and other Android™-based devices. It should be appreciated that other types of computing devices capable of connecting to and/or interacting with the components of system100can be used without departing from the scope of invention. AlthoughFIG.1depicts a single mobile computing device102, it should be appreciated that system100can include a plurality of mobile computing devices.

Mobile computing device102is configured to receive a data representation of a live audio signal corresponding to the live event via wireless network106. For example, in some embodiments, mobile computing device102is configured to receive the data representation of the live audio signal corresponding to the live event from server computing device104via wireless network106, where server computing device104is coupled to an audio source at the live event (e.g., a soundboard that is capturing live audio). Mobile computing device102is also configured to process the data representation of the live audio signal into a live audio stream.

Mobile computing device102is also configured to initiate playback of the live audio stream via a first headphone (not shown) communicatively coupled to the mobile computing device102at the live event. For example, the user of mobile computing device102can connect a headphone to the device via a wired connection (e.g., by plugging the headphone into a jack on the mobile computing device) or via a wireless connection (e.g., pairing the headphone to the mobile computing device via a short-range communication protocol such as Bluetooth™). Mobile computing device102can then initiate playback of the live audio stream via the headphone.

FIG.2is a flow diagram of a computerized method200of real-time detection of millisecond gaps in live audio streams, according to an illustrative embodiment of the invention. In some embodiments, the method200ofFIG.2can be performed by system100ofFIG.1.

Mobile computing device102receives (step202) a data representation of a live audio signal corresponding to the live event from server computing device102via wireless network106. In some embodiments, application110of mobile computing device102is configured to activate a function to establish a connection to server computing device104via wireless network106in order to begin receiving the data representation of the live audio signal. An exemplary application110can be an app downloaded to and installed on the mobile computing device102via, e.g., the Apple® App Store or the Google® Play Store. A user of mobile computing device102can launch application110and interact with one or more user interface elements displayed by the application110on a screen of the mobile computing device102to start receiving the data representation of the live audio signal.

Mobile computing device102records (step204) the data representation of the live audio signal to a file. In some embodiments, mobile computing device102can create an audio file in local memory and store at least a portion of the data representation of the live audio signal in the audio file. Exemplary file formats for storing the data representation of the live audio signal include, but are not limited to, WAV, MP3, OGG, FLAC, Apple® Lossless, and AAC. In some embodiments, mobile computing device102can store the data representation of the live audio signal in a temporary buffer location in memory.

Mobile computing device102converts (step206) the audio file to an array of samples. In some embodiments, mobile computing device102converts the audio file into a NumPy array using one or more Python libraries/modules, such as PyAudio (available at people.csail.mit.edu/hubert/pyaudio), SciPy.io.wavefile (available in v.1.31.1 of scipy), or Pydub (available at pypi.org/project/pydub/).

Mobile computing device102groups (step208) samples from the array of samples into audio frames. In some embodiments, mobile computing device102can determine a frame length of the audio frames based upon, e.g., a desired level of time resolution and/or accuracy. It should be appreciated that a low frame length improves time resolution by sacrificing accuracy. In some embodiments, frame length is measured in seconds or milliseconds.

Mobile computing device102calculates (step210) the root mean square (RMS) and spectral bandwidth (SBW) of each audio frame. Generally, RMS is used to measure the average loudness of a waveform and SBW is used to measure the spread or width of frequencies present in an audio signal. In some embodiments, mobile computing device102uses the librosa.feature.rms function (in the librosa v0.10.2 Python package, available at librosa.org) to calculate the RMS for each audio frame. In some embodiments, mobile computing device102uses the librosa.feature.spectral_bandwidth function (also in the librosa v0.10.2 Python package) to calculate the SBW for each audio frame.

Mobile computing device102marks (step212) an audio frame as an error if the RMS or SBW measurements for the audio frame (as calculated above) fall below a threshold value. In some embodiments, the RMS measurement for an audio frame can be below-30 dB-indicating that the frame is empty or received with an error. In some embodiments, the SBW measurement for an audio frame can be between 0 and 10 hZ-indicating that there is almost no spectral spread in the given frame and suggesting that the audio frame is empty or received with an error.

Mobile computing device102passes (step214) the list of errors to a Hidden Markov Model (HMM) to determine the probabilities of error states. Generally, an HMM is a probabilistic model that consists of a sequence of hidden states (which are not directly observable), each of which generates an observation. The goal of the HMM is to estimate the sequence of hidden states based on a sequence of observations. In some embodiments, mobile computing device102uses the hmmlearn Python package (available at github.com/hmmlearn/hmmlearn) with the marked error audio frames as input to determine the probabilities that the error audio frames are indicative of a probable error state.

Mobile computing device102passes (step216) the list of errors to an algorithm that determines the parameters of a Gilbert-Elliot model that generates the given list of errors. Generally, a Gilbert-Elliot model is a two-state model for describing errors in a digital communication link (as described in J. Pieper and S. Voran, “Relationships Between Gilbert-Elliot Burst Error Model Parameters and Error Statistics,” U.S. Dept. of Commerce NTIA Technical Memorandum 23-565, January 2023, which is incorporated herein by reference). In some embodiments, mobile computing device102uses the gilbert-elliot-model Python package (available at github.com/NTIA/gilbert-elliot-model) to estimate model parameters for a Gilbert-Elliot model.

Mobile computing device102maps (step218) the Gilbert-Elliot models to a mean opinion score (MOS) of similar error models. In some embodiments, mobile computing device102maps the Gilbert-Elliot models to a MOS of similar error models in order to evaluate the overall quality of the streamed audio. Further detail regarding the process for mapping the Gilbert-Elliot models to a MOS is described in (i) G. Haßlinger and Oliver Hohlfeld, “The Gilbert-Elliot Model for Packet Loss in Real Time Services on the Internet,”Proceedings of the14thGI/ITG Conference on Measurement, Modelling and Evaluation of Computer and Communication Systems(MMB2008), Mar. 31 to Apr. 2, 2008, Dortmund Germany, pp. 1-15; and (ii) O. Hohlfeld et al., “Packet Loss in Real-Time Services: Markovian Models Generating QoE Impairments,” 2008 16th International Workshop on Quality of Service, Enschede, Netherlands, 2008, pp. 239-248; each of which is incorporated herein by reference.

FIGS.3A-Dare diagrams of exemplary audio samples analyzed using system architecture200, according to an illustrative embodiment of the invention. The exemplary audio sample is of a fifteen-second duration. Using system architecture200, the Gilbert-Elliot model was applied using different error rates, and expected burst duration, to the original audio sample. The audio and errors are visualized under the audio clip. Because the Gilbert-Elliot model is stochastic, the specified error parameters do not always yield the desired error pattern so the actual error statistics are printed out under each graph for transparency. For example,FIG.3Aillustrates exemplary audio with approximately 10% loss at 300 ms (14400 samples) burst duration.FIG.3Billustrates exemplary audio with approximately 10% loss at 200 ms (9600 samples) burst duration.FIG.3Cillustrates exemplary audio with approximately 10% loss at 100 ms (4800 samples) burst duration.FIG.3Dillustrates exemplary audio with approximately 10% loss at 50 ms (2400 samples) burst duration.

FIG.4is a schematic flow diagram of a process400for real-time detection of millisecond gaps in live audio streams using system architecture200, according to an illustrative embodiment of the invention. Process400begins by receiving, by a first mobile computing device102, a data representation of a live audio signal corresponding to a live event via a wireless network106at step402. In some embodiments, process400begins by receiving, by the first mobile computing device102, the data representation of the live audio signal corresponding to the live event from an audio server computing device104via the wireless network106. Process400continues by processing, by the first mobile computing device102, the data representation of the live audio signal corresponding to the live event into a live audio stream at step404.

Process400continues by transmitting, by the first mobile computing device102, the live audio stream to a second mobile computing device102communicatively coupled to the first mobile computing device102at step406. For example, in some embodiments, the first mobile computing device102and the second mobile computing device102are communicatively coupled via a wired connection. In other embodiments, the first mobile computing device102and the second mobile computing device102are communicatively coupled via a wireless connection.

Process400continues by detecting, by the second mobile computing device102, a first gap in the received live audio stream at step408. Process400finishes by calculating, by the second mobile computing device102, at least one of a timestamp corresponding to the first gap in the received live audio stream or a duration of the first gap in the received live audio stream at step410. In some embodiments, the duration of the first gap in the received live audio stream is less than 20 milliseconds. For example, in some embodiments, the duration of the first gap in the received live audio stream is less than 10 milliseconds. In some embodiments, the duration of the first gap in the received live audio stream is less than 5 milliseconds.

In some embodiments, process400continues by detecting, by the second mobile computing device102, a second gap in the received live audio stream. For example, in some embodiments, process400continues by calculating, by the second mobile computing device102, at least one of a timestamp corresponding to the second gap in the received live audio stream or a duration of the second gap in the received live audio stream. In some embodiments, process400continues by calculating, by the second mobile computing device102, a rate of gaps corresponding to the received live audio stream.

Process400can be implemented using a system for real-time detection of millisecond gaps in live audio streams. The system includes a first mobile computing device102communicatively coupled to a second mobile computing device102. For example, in some embodiments, the first mobile computing device102and the second mobile computing device102are communicatively coupled via a wired connection. In other embodiments, the first mobile computing device102and the second mobile computing device102are communicatively coupled via a wireless connection.

The first mobile computing device102is configured to receive a data representation of a live audio signal corresponding to a live event via a wireless network106. In some embodiments, the system includes an audio server computing device104communicatively coupled to the first mobile computing device102, the first mobile computing device102configured to receive the data representation of the live audio signal corresponding to the live event from the audio server computing device104via the wireless network106.

The first mobile computing device102is also configured to process the data representation of the live audio signal corresponding to the live event into a live audio stream. The first mobile computing device102is also configured to transmit the live audio stream to the second mobile computing device102communicatively coupled to the first mobile computing device.

The second mobile computing device102is configured to detect a first gap in the received live audio stream. The second mobile computing device102is also configured to calculate at least one of a timestamp corresponding to the first gap in the received live audio stream or a duration of the first gap in the received live audio stream. In some embodiments, the duration of the first gap in the received live audio stream is less than 20 milliseconds. For example, in some embodiments, the duration of the first gap in the received live audio stream is less than 10 milliseconds. In some embodiments, the duration of the first gap in the received live audio stream is less than 5 milliseconds.

In some embodiments, the second mobile computing device102is also configured to detect a second gap in the received live audio stream. For example, in some embodiments, the second mobile computing device102is also configured to calculate at least one of a timestamp corresponding to the second gap in the received live audio stream or a duration of the second gap in the received live audio stream. In some embodiments, the second mobile computing device102is also configured to calculate a rate of gaps corresponding to the received live audio stream.

Upon detecting one or more gaps in the received live audio stream and determining the duration and/or rate of the gaps, the second mobile computing device102can be configured to transmit a notification message to a remote computing device (such as first mobile computing device102and/or server computing device104). In some embodiments, the notification message can comprise data and/or metadata associated with the detected gaps-such as stream statistics, timestamps, gap durations, gap rates, and other data. The remote computing device can receive the notification message and implement one or more audio transmission corrections to automatically reduce or eliminate gaps in the stream. For example, the remote computing device can increase a buffer size for storing the live audio stream, e.g., to ensure that buffer underruns do not occur or are less likely to occur.

The systems and methods described herein can be implemented using supervised learning and/or machine learning algorithms. Supervised learning is the machine learning task of learning a function that maps an input to an output based on example of input-output pairs. It infers a function from labeled training data consisting of a set of training examples. Each example is a pair consisting of an input object and a desired output value. A supervised learning algorithm or machine learning algorithm analyzes the training data and produces an inferred function, which can be used for mapping new examples.

The computer program can be deployed in a cloud computing environment (e.g., Amazon® AWS, Microsoft® Azure, IBM® Cloud™). A cloud computing environment includes a collection of computing resources provided as a service to one or more remote computing devices that connect to the cloud computing environment via a service account-which allows access to the aforementioned computing resources. Cloud applications use various resources that are distributed within the cloud computing environment, across availability zones, and/or across multiple computing environments or data centers. Cloud applications are hosted as a service and use transitory, temporary, and/or persistent storage to store their data. These applications leverage cloud infrastructure that eliminates the need for continuous monitoring of computing infrastructure by the application developers, such as provisioning servers, clusters, virtual machines, storage devices, and/or network resources. Instead, developers use resources in the cloud computing environment to build and run the application and store relevant data.

Method steps can be performed by one or more processors executing a computer program to perform functions of the invention by operating on input data and/or generating output data. Subroutines can refer to portions of the stored computer program and/or the processor, and/or the special circuitry that implement one or more functions. Processors suitable for the execution of a computer program include, by way of example, special purpose microprocessors specifically programmed with instructions executable to perform the methods described herein, and any one or more processors of any kind of digital or analog computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and/or data. Exemplary processors can include, but are not limited to, integrated circuit (IC) microprocessors (including single-core and multi-core processors). Method steps can also be performed by, and an apparatus can be implemented as, special purpose logic circuitry, e.g., a FPGA (field programmable gate array), a FPAA (field-programmable analog array), a CPLD (complex programmable logic device), a PSoC (Programmable System-on-Chip), ASIP (application-specific instruction-set processor), an ASIC (application-specific integrated circuit), Graphics Processing Unit (GPU) hardware (integrated and/or discrete), another type of specialized processor or processors configured to carry out the method steps, or the like.

Memory devices, such as a cache, can be used to temporarily store data. Memory devices can also be used for long-term data storage. Generally, a computer also includes, or is operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. A computer can also be operatively coupled to a communications network in order to receive instructions and/or data from the network and/or to transfer instructions and/or data to the network. Computer-readable storage mediums suitable for embodying computer program instructions and data include all forms of volatile and non-volatile memory, including by way of example semiconductor memory devices, e.g., DRAM, SRAM, EPROM, EEPROM, and flash memory devices (e.g., NAND flash memory, solid state drives (SSD)); magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and optical disks, e.g., CD, DVD, HD-DVD, and Blu-ray disks. The processor and the memory can be supplemented by and/or incorporated in special purpose logic circuitry.

To provide for interaction with a user, the above-described techniques can be implemented on a computing device in communication with a display device, e.g., a CRT (cathode ray tube), plasma, or LCD (liquid crystal display) monitor, a mobile device display or screen, a holographic device and/or projector, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse, a trackball, a touchpad, or a motion sensor, by which the user can provide input to the computer (e.g., interact with a user interface element). The systems and methods described herein can be configured to interact with a user via wearable computing devices, such as an augmented reality (AR) appliance, a virtual reality (VR) appliance, a mixed reality (MR) appliance, or another type of device. Exemplary wearable computing devices can include, but are not limited to, headsets such as Meta™ Quest 3™ and Apple® Vision Pro™ Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, and/or tactile input.

The components of the computing system can be interconnected by transmission medium, which can include any form or medium of digital or analog data communication (e.g., a communication network). Transmission medium can include one or more packet-based networks and/or one or more circuit-based networks in any configuration. Packet-based networks can include, for example, the Internet, a carrier internet protocol (IP) network (e.g., local area network (LAN), wide area network (WAN),), a private IP network, an IP private branch exchange (IPBX), a wireless network (e.g., radio access network (RAN), Bluetooth™, near field communications (NFC) network, Wi-Fi™, WiMAX™, general packet radio service (GPRS) network, HiperLAN), and/or other packet-based networks. Circuit-based networks can include, for example, the public switched telephone network (PSTN), a legacy private branch exchange (PBX), a wireless network (e.g., RAN, code-division multiple access (CDMA) network, time division multiple access (TDMA) network, global system for mobile communications (GSM) network), cellular networks, and/or other circuit-based networks.

The methods and systems described herein can utilize artificial intelligence (AI) and/or machine learning (ML) algorithms to process data and/or control computing devices. In one example, a classification model, is a trained ML algorithm that receives and analyzes input to generate corresponding output, most often a classification and/or label of the input according to a particular framework.

While the invention has been particularly shown and described with reference to specific preferred embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the following claims. One skilled in the art will realize the subject matter may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting the subject matter described herein.