Downloadable audio features

A method of providing custom audio processing in an audio device having an embedded audio DSP framework is disclosed. The audio device has an associated application that runs on a personal computing device. The method comprises receiving a notification of the availability of an audio processing configuration file, the audio processing configuration file identifying audio modules, interconnections between the audio modules and related settings for custom audio processing, transmitting a request for the audio processing configuration file from the personal computing device, receiving the audio processing configuration file at the personal computing device, transmitting the audio processing configuration file to the audio device from the personal computing device, and loading the audio processing configuration file into the audio device.

BACKGROUND

Current audio devices, such as headphones, earbuds, smart speakers, televisions and so forth, have become more sophisticated and connected. Consumers have also become more demanding. Current solutions for managing audio playback typically include the ability to adjust the volume and check the battery level of an audio rendering device using an app on an associated smartphone. In some instances, other settings or functions may be performed by an associated app, such as activating noise canceling or updating firmware.

DETAILED DESCRIPTION

In some examples, provided is a method of providing custom audio processing in an audio device. The method includes receiving, at an application associated with the audio device that runs on a personal computing device, a notification of the availability of an audio processing configuration file. The audio processing configuration file identifying audio modules, interconnections between the audio modules and related settings for custom audio processing. The method further comprises transmitting a request for the audio processing configuration file from the personal computing device, receiving the audio processing configuration file at the personal computing device, transmitting the audio processing configuration file to the audio device from the personal computing device, and loading the audio processing configuration file into the audio device.

The audio device may include an audio DSP processing framework including a run-time execution engine and run-time libraries, and the audio processing configuration file may be a binary file that configures specific run-time libraries to provide customized audio processing. The audio processing configuration file may have been created by an independent third-party developer and not a manufacturer of the device.

The method may further include providing a user notification of the availability of an audio processing configuration file, and receiving user input via the application on the personal computing device to load the audio processing configuration file into the audio device. Additionally, the method may include accessing an auto-update setting for the application or the personal computing device, and transmitting the request for the audio processing configuration file from the personal computing device without user input, based on the auto-update setting being positive. The audio processing configuration is loaded into the audio device without requiring updating or recompiling of executable code.

In some examples, provided is a non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations for providing custom audio processing in an audio device according to the methods described above, the operations including but not being limited to receiving, at an application associated with the audio device that runs on a personal computing device, a notification of the availability of an audio processing configuration file, the audio processing configuration file identifying audio modules, interconnections between the audio modules and related settings for custom audio processing, transmitting a request for the audio processing configuration file from the personal computing device, receiving the audio processing configuration file at the personal computing device, transmitting the audio processing configuration file to the audio device from the personal computing device, and loading the audio processing configuration file into the audio device.

In some examples, provided is computing apparatus comprising a processor and a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for providing custom audio processing in an audio device according to the methods described above, the operations the operations including but not being limited to receiving, at an application associated with the audio device that runs on a personal computing device, a notification of the availability of an audio processing configuration file, the audio processing configuration file identifying audio modules, interconnections between the audio modules and related settings for custom audio processing, transmitting a request for the audio processing configuration file from the personal computing device, receiving the audio processing configuration file at the personal computing device, transmitting the audio processing configuration file to the audio device from the personal computing device, and loading the audio processing configuration file into the audio device.

FIG.1illustrates example wireless ear buds100that together form a set of wearable audio devices. Each wireless ear bud102includes a communication interface108used to communicatively couple with an audio source or sink device, e.g., a client device206(seeFIG.2) that can provide audio data that the wireless ear buds100can reproduce as audio signals for a user of the wireless ear buds100, or that can receive audio data from the wireless ear buds100. Each wireless ear bud102also includes a battery116and optionally one or more sensors104for detecting a wearing status of the wireless ear buds100, e.g., when a wireless ear bud102is placed in or on and/or removed from an ear.

Additionally, each wireless ear bud102includes an audio transducer106for converting a received signal including audio data, into audible sound and one or more microphones118for generating ambient and speech signals. A receive audio signal can be received from a paired companion communication device such as client device206via the communication interface108, or alternatively the receive signal may be relayed from one wireless ear bud102to the other. A transmit audio signal can be generated from the one or more microphones118in the wireless ear buds100.

One or both of the wireless ear buds102include a DSP framework112for processing received audio signals and/or signals from the one or more microphones118, to provide to the audio transducer106or a remote user. The DSP framework112is a software stack running on a physical DSP core (not shown) or other appropriate computing hardware, such as a networked processing unit, accelerated processing unit, a microcontroller, graphics processing unit or other hardware acceleration. The DSP core will have additional software such as an operating system, drivers, services, and so forth. One or both of the wireless ear bud102also include a processor110and memory114. The memory114in the wireless ear buds100stores firmware for operating the wireless ear buds100and for pairing the wireless ear buds100with companion communication devices.

Although described herein with reference to wireless ear buds, it will be appreciated that the methods and structures described herein are applicable to any audio device that may benefit therefrom.

FIG.2illustrates a system200in which a server204, a client device206and a developer device208are connected to a network202.

In various embodiments, the network202may include the Internet, a local area network (“LAN”), a wide area network (“WAN”), and/or other data network. In addition to traditional data-networking protocols, in some embodiments, data may be communicated according to protocols and/or standards including near field communication (“NFC”), Bluetooth, power-line communication (“PLC”), and the like. In some embodiments, the network202may also include a voice network that conveys not only voice communications, but also non-voice data such as Short Message Service (“SMS”) messages, as well as data communicated via various cellular data communication protocols, and the like.

In various embodiments, the client device206may include desktop PCs, mobile phones, laptops, tablets, wearable computers, or other computing devices that are capable of connecting to the network202and communicating with the server204, such as described herein. The client device206may be paired with wireless ear buds100(or other audio devices) that provide audio output to a user of the client device206. Additionally, one or more developer devices208may be utilized to generate downloadable binary files that may be used to customize the audio of the wireless ear buds100as will be discussed in more detail below.

In various embodiments, additional infrastructure (e.g., short message service centers, cell sites, routers, gateways, firewalls, and the like), as well as additional devices may be present. Further, in some embodiments, the functions described as being provided by some or all of the server204and the client device206may be implemented via various combinations of physical and/or logical devices. However, it is not necessary to show such infrastructure and implementation details inFIG.2in order to describe an illustrative embodiment.

FIG.3illustrates a development and implementation flow diagram300according to some examples. Shown in the figure are a developer device208, a binary file302and a DSP framework112.

The developer device208hosts an application for use in developing downloadable custom audio features for the DSP framework112. As user herein, the term custom includes both new audio features and updates to existing audio features. The application may for example be a windows-based graphical design environment that permits the definition of audio processing pathways by assembling and interconnecting various audio processing blocks and specifying associated parameters and interconnections. The final output of the application is the binary file302.

The binary file302is a configuration file for the data-driven DSP framework112that specifies which audio modules in the DSP framework112to use (e.g., from standard module pack316, advanced module pack318and/or voice module pack320), how they are interconnected (via audio pipeline manager314), and which parameter settings or values (not shown) are employed to provide custom audio processing. The binary file302can also impact the usage of input and control signals, such as the pressing of an existing button on the wireless ear buds100, or a control signal from client device206or application402, could be incorporated into the different audio processing functions as specified by the binary file302. The binary file302can define processing that operates on incoming audio, outgoing audio, and can also define processing internally to the DSP framework112to generate or extract parameter values from the input, output or intermediate audio, which values can be used for audio or other system functions. In one example, the binary file is a netlist.

The binary file302is a platform independent data file that is stored in the DSP framework112and that configures the run-time execution engine304in the DSP framework112. Since the binary file302defines audio processing that takes place within the DSP framework112and does not include executable code as such, it is not possible for a developer to include malicious executable code that can operate on non-audio information in the wireless ear buds or client device206. Since the processing specified by the binary file302is limited to interconnections of and parameters for the audio modules in the DSP framework112, privacy concerns are reduced when providing custom audio processing, since it is not possible for the DSP framework112to “reach out” and capture personally identifiable information such as biometric, financial or device positional information, or audio call data identifying called or calling parties.

Conventional audio customization on the other hand would either typically only permit the provision of custom “tuning” parameters that can adjust existing audio processing, or requires a firmware update for each audio device, in which and audio algorithms would have to be built specifically for each audio device and processor. An original hardware manufacturer would thus either have to curate audio features and embed them itself, or open up its firmware to third parties, allowing others to rebuild or update the firmware, which is unlikely to occur. Additionally, updates done using DLLs would require the recompilation and linking of any associated code.

The DSP framework112as embedded in the wireless ear buds100or (other audio product) will include core audio processing defined by the manufacturer of the wireless ear buds100. The DSP framework112also provides a “toolbox” of audio processing modules that can be interconnected by a binary file302to provide additional or alternative processing to the core audio processing provided by the manufacturer. Third party or other application developers can, at the users discretion, augment or scale the audio processing of released products using custom binary files302provided as disclosed herein. The binary files302permit the dynamic updating of the audio signal chain completely separately from the operating system of the audio device or applications on the audio device, and without updating or relinking any executable code.

The DSP framework112is an embedded audio processing engine that includes optimized target-specific libraries for various audio processing applications. The DSP framework112is available for multiple processors and supports multicore and multi-instance implementations. In some examples, the DSP framework112includes a run-time execution engine304that can be controlled by multiple API's such as a tuning API306, an audio API308, and a control API310. The112also includes a memory manager312and an audio pipeline manager314. The DSP framework112also includes various audio processing module packs, such as a standard module pack316, advanced module pack318and a voice module pack320. As indicated by the dashed outline for the voice module pack320, the exact number and types of different packs will depend on the particular implementation.

FIG.4illustrates a development and implementation flow diagram400according to some examples. Shown in the figure are wireless ear buds100, a developer device208, a binary file302, a DSP framework112and a client device206hosting an application402.

As illustrated, the developer of an audio device (for example the wireless ear buds100) embeds the DSP framework112in the audio device prior to release. The DSP framework112will include appropriate modules for the particular audio device, including the standard module pack316, the advanced module pack318and optionally, additional module packs such as the voice module pack320.

As discussed above, the developer device208hosts an application for use in developing downloadable custom audio features for the DSP framework112. The developer device208is used by an application developer that has or is developing the application402, which can access or control the wireless ear buds100. In some examples, the application402may be an audio app such as a music or video player or an integrated virtual assistant. In other examples, the developer of the wireless ear buds100may have an application402that a user of the client device206can use to access or manage the wireless ear buds100, in which case the developer of the audio device and the developer of the application402may be one and the same. In such a case, third parties may be permitted to provide custom audio processing to the wireless ear buds100via an application402provided by the manufacturer of the wireless earbuds. Similarly, the custom audio processing may be defined by an independent audio processing developer or vendor to improve the audio processing of the wireless ear buds100when the earbuds are used with the application402itself.

That is, the manufacturer of the wireless ear buds100may be one party, the provider of the application may be a different (third) party, and the developer of the custom audio processing may again be a different (third) party. So in one example, the wireless ear buds100are sold by their manufacturer, the application402is a video streaming application provided by a streaming service provider, and the developer is a custom audio developer who creates a binary file302that provides custom audio processing for the wireless ear buds100when a user is watching video that is streamed using the application402.

The developer device208is used by the app developer or third-party audio processing developer to create a binary file302to provide custom audio processing to the wireless ear buds100. As mentioned above, the binary file302is a configuration file for the data-driven DSP framework112that tells the embedded target which audio modules in the DSP framework112to use, how they are interconnected, and which parameter settings to employ. The binary file302can for example provide more sophisticated or different audio processing than what was in the original DSP framework112when it was embedded in the wireless ear buds100originally. The binary file302can also impact the usage of input and control signals, such as the pressing of an existing button on the wireless ear buds100, or a control signal from client device206or application402, could be incorporated into the different audio processing functions as specified by the binary file302. The binary file302is uploaded from the developer device208to a server204that is accessible to the client device206.

The binary file302is then downloaded into the application402on the client device206. This can be done directly over the network202from a server204operated by the app developer or the binary file302can be uploaded to an app store, where it can be downloaded by the user of the client device206in response to prompts in the application402or when updating the application402when prompted to do so by the OS on the client device206.

FIG.5illustrates a flowchart500for providing customized audio processing according to some examples. The flowchart500commences at operation502with creation of the binary file302by a developer of customized audio processing. This is done for example on a developer device208that hosts an application for use in developing downloadable custom audio features for the DSP framework112. The application may for example be a windows-based graphical design environment that permits the definition of audio processing pathways by assembling and interconnecting various audio processing blocks and specifying associated parameters and interconnections.

The binary file302is then made available to the application402at operation504. This can be done by uploading the binary file302from the developer device208to a server204that is accessible to the client device206. For example, the binary file302can be uploaded to an app store, where it can be downloaded by the user of the client device206in response to prompts in the application402or when updating the application402if prompted to do so by the OS of the client device206.

At operation506, the application402or client device206is notified of the availability of the binary file302. Depending on the settings on or implementation of the application402or client device206, downloading of the binary file302may require that a notification and acceptance of the binary file be indicated by a user of the client device206. Such a notification or prompt informs the user of the availability of custom audio processing for the wireless ear buds100(or other audio device) and prompts the user to select an option to download the binary file302.

In response to receiving user input (if required) to download the binary file302in operation508(that is, to update the wireless ear buds100with custom audio processing or to update the application402itself), the application402or client device206transmits a request to the server to receive the binary file302in operation510. In response, the server204transmits the binary file302to the client device206in operation512. The binary file302is received by the client device206and application402in operation514, over network202from server204.

Alternatively, in the event that automatic updates are permitted or have been authorized for the application402or client device206or wireless ear buds100, in response to receiving a corresponding instruction from the client device206, the server204transmits the binary file302to the client device206in operation512. An auto-update setting in the application402or on the client device206can be checked to determine whether or not automatic updates are permitted.

The binary file302is received by the client device206and application402in operation514, over network202from server204. The binary file302may also be provided as an update to the application402.

The binary file302is then transmitted to the audio device in (such as the wireless ear buds100) from the client device206by the application402in operation516, typically over a wireless data link such as Bluetooth. The binary file302is then loaded into the DSP framework112by the audio device in operation518. Customized audio processing can then be provided by the audio device according to contents of the binary file302, which specifies which audio modules in the DSP framework112to use, how they are interconnected, and which parameter settings to employ to provide the custom audio processing. Selection of audio processing options and parameters can be done from the application402via the APIs provided by the DSP framework112.

FIG.6illustrates a diagrammatic representation of a machine600in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein, according to an example embodiment. Specifically,FIG.6shows a diagrammatic representation of the machine600in the example form of a computer system, within which instructions608(e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine600to perform any one or more of the methodologies discussed herein may be executed. For example the instructions608may cause the machine600to execute the methods described above. The instructions608transform the general, non-programmed machine600into a particular machine600programmed to carry out the described and illustrated functions in the manner described. In alternative embodiments, the machine600operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine600may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine600may comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions608, sequentially or otherwise, that specify actions to be taken by the machine600. Further, while only a single machine600is illustrated, the term “machine” shall also be taken to include a collection of machines600that individually or jointly execute the instructions608to perform any one or more of the methodologies discussed herein.

The machine600may include processors602, memory604, and I/O components642, which may be configured to communicate with each other such as via a bus644. In an example embodiment, the processors602(e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an ASIC, a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor606and a processor610that may execute the instructions608. The term “processor” is intended to include multi-core processors that may comprise two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously. AlthoughFIG.6shows multiple processors602, the machine600may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiples cores, or any combination thereof.

The memory604may include a main memory612, a static memory614, and a storage unit616, both accessible to the processors602such as via the bus644. The main memory604, the static memory614, and storage unit616store the instructions608embodying any one or more of the methodologies or functions described herein. The instructions608may also reside, completely or partially, within the main memory612, within the static memory614, within machine-readable medium618within the storage unit616, within at least one of the processors602(e.g., within the processor's cache memory), or any suitable combination thereof, during execution thereof by the machine600.

In further example embodiments, the I/O components642may include biometric components632, motion components634, environmental components636, or position components638, among a wide array of other components. For example, the biometric components632may include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion components634may include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. The environmental components636may include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detection concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components638may include location sensor components (e.g., a GPS receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies. The I/O components642may include communication components640operable to couple the machine600to a network620or devices622via a coupling624and a coupling626, respectively. For example, the communication components640may include a network interface component or another suitable device to interface with the network620. In further examples, the communication components640may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices622may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

The various memories (i.e., memory604, main memory612, static memory614, and/or memory of the processors602) and/or storage unit616may store one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., the instructions608), when executed by processors602, cause various operations to implement the disclosed embodiments.