Flexible circuit board design in a brain computer interface module

A flexible printed circuit board (FPCA) of a brain computer interface (BCI) module is configured to interconnect a plurality of emitter assemblies and a plurality of detector assemblies of the BCI module. The FPCA comprises a connector portion for connecting the FPCA to a controller of the BCI module, a plurality of rigid sections, a plurality of flexible sections. A first subset of rigid sections is configured to mount the plurality of emitter assemblies. A second subset of rigid sections is configured to mount the plurality of detector assemblies. Each flexible section is configured to attach the two or more rigid sections of the plurality of rigid sections to each other. The plurality of flexible sections allows the plurality of emitter assemblies and the plurality of detector assemblies to stretch to conform to a head of a user.

BACKGROUND

This disclosure relates generally to brain computer interface systems, and specifically to a wearable brain computer interface system with an increased dynamic range sensor.

Communication via physical actions, such as textual entry or manipulation of a user interface on a mobile or other device is a key form of interaction amongst individuals today. Additionally, certain online systems, such as online social networks, thrive on the network of users that frequent the online social network on a consistent basis. One component of online social networks is the ability of a user to interact with objects (e.g., electronically provided content) in an online or virtual setting. In many scenarios, detection of interactions requires the user to type or enter words and phrases through a physical means (e.g., a keyboard or clicking on a virtual keyboard) and/or to audibly provide commands. Physically entering words and phrases or providing audible commands may be cumbersome or impossible for certain individuals. Additionally, and more generally, physical entry of words and phrases for all individuals is often an inefficient way to communicate, as typing or otherwise manipulating various user interfaces can be cumbersome.

Brain computer interface (BCI) systems are being explored in relation to some of these problems. However, traditional brain computer interface (BCI) systems typically implement electrical signal detection methods to characterize brain activity. Such systems are typically used in clinical or academic settings, and often are not designed for use by users during their normal daily lives. In relation to user factors, such systems often lack features that allow users to properly position sensing components in a repeatable and reliable manner, as well as to maintain contacts between sensing components and desired body regions as a user moves throughout his or her daily life. Miniaturization of such BCI systems also provides challenges.

Other types of BCI systems use optical signals to characterize brain activity. Conventional optical wearable and non-wearable BCI systems implement fiber optics to couple light from the scalp to detectors and emitters. However, such systems are often bulky and inhibit a user's ability to perform other daily tasks. Accordingly, there exists a need for a compact BCI system that can be easily worn by a user.

Additionally, fields exploring other sensing regimes for detection and decoding of brain activity are nascent, and traditional sensors used for other sensing regimes have insufficient dynamic range and often provide limitations in readout speed, thereby limiting their use in applications where rapid decoding of brain activity is important.

SUMMARY

Disclosed herein are systems and methods for enabling a user to communicate using a brain computer interface (BCI) system through unspoken communications. As used hereafter, unspoken methods and/or unspoken communications refer to communications that can be performed by an individual through non-verbal (e.g., without verbal sounds), non-physical (e.g., not inputted by an individual through a physical means such as a keyboard, mouse, touchscreen, and the like), and/or non-expressive (e.g., not expressed through facial features, body language, and the like) means.

Generally, a BCI module interprets an individual's brain activity to characterize intentions of the individual in interacting with content in the environment of the user. In particular embodiments, the BCI module includes a flexible printed circuit assembly (FPCA) configured to conform to a head of a user, at least one emitter package, at least one detector package, and a controller. The emitter package includes at least one emitter configured to emit light towards the head of the user and the detector package includes at least one detector configured to detect light reflected from the head. Each emitter or detector of the BCI module is encased in a ferrule to direct light towards the head of the user and towards the detector. The controller is configured with electronics configured to provide power and/or computing functionality. The BCI module is configured in a wearable form factor that allows the user to repeatedly and reliably position light transmitting and light sensing components at the body region. As such, the system can include components appropriate for a small form factor that is portable and worn discreetly at a head region of the user.

Embodiments also relate to a sensor system for a brain computer interface (BCI) that enables detection and decoding of brain activity by optical tomography. The sensor system includes an array of pixels arranged as grouped pixel units to provide increased dynamic range. One or more of the grouped pixel units can operate in a saturated mode while providing information useful for decoding brain activity. Furthermore, the grouped pixel units are arranged to enable fast readout by a controller, thereby increasing detection and decoding ability by systems implementing the sensor design. The grouped pixel units of the sensor system are aligned with optical fibers of an interface to a body region of a user, where the optical fibers can be retained in position relative to the grouped pixel units by an optically transparent substrate that provides mechanical support while minimizing factors associated with divergence of light transmitted through optical fibers.

Embodiments also relate to a ferrule assembly in which a ferrule is mounted on a rigid section of a circuit board. An optical package including an optical element and a controller configured to operate the optical element are also mounted on the rigid section of the circuit board. Examples of optical elements may include an emitter, a detector, or a combination thereof. The ferrule is configured to optically couple to electronics for detecting brain signals, wherein the ferrule has optical properties to guide light between electronics and a user's head.

Embodiments of a flexible printed circuit assembly (FPCA) of a BCI module are configured to interconnect at least one emitter package to at least one detector package of the BCI module. The FPCA is comprised of rigid sections, each of which is configured to mount both emitter and detector package, and flexible sections, each of which is configured to connect at least two of the rigid sections. The geometry of the flexible sections enables the flexible sections to stretch to conform to the head of a user. The connection of the flexible sections to rigid sections of the FPCA enables the FPCA as a whole to stretch to confirm to the head of a user.

Embodiments also relate to decoding architecture that rapidly (e.g., in real time or near real time) decodes light-derived signals to extract predicted user actions or intents (e.g., commands) in relation to interactions with objects (e.g., virtual objects, physical objects), such that the user can manipulate the objects or otherwise receive assistance without manually interacting with an input device (e.g., touch input device, audio input device, etc.). The decoding architecture thus enables a neural decoding process with a neural signal stream as an input, and provides feedback to the user, where the feedback is used to train the neural decoding algorithm and user behavior. The neural signals can be blood oxygenation level dependent (BOLD) signals associated with activation of different articulators of the motor cortex, and signals can characterize both actual and imagined motor cortex-related behaviors. With training of the decoding algorithm, rapid calibration of the BCI for new users can additionally be achieved.

It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. For example, a letter after a reference numeral, such as “150a,” indicates that the text refers specifically to the element having that particular reference numeral. A reference numeral in the text without a following letter, such as “150,” refers to any or all of the elements in the figures bearing that reference numeral (e.g. “computing component150” in the text refers to reference numerals “computing component150a” and/or “computing component150b” in the figures).

DETAILED DESCRIPTION

Embodiments relate to a brain computer interface (BCI) including an interface module that transmits light from an emitter assembly to a body region of a user and receives light signals reflected from the body region of the user. The interface module additionally includes a controller that provides power to both the emitter and detector assemblies and computing functionality to covert the received light signals into one or more signals representing the user's brain activity. The BCI interface module is designed to be worn at a head region of a user and to generate optical signals that can be used to characterize brain activity of the user, where decoded brain activity can be used as inputs to control other systems and/or electronic content provided to the user.

In relation to brain activity sensing, embodiments also relate to an emitter array of the BCI, where the emitter array is provided in a miniaturized form factor that outputs light with appropriate characteristics to enable measurement of blood oxygenation by the detector array, where oxygenation levels can be determined relative to a reference state. In some embodiments, a detector array integrated into the BCI module is configured to measure other types of optical brain signals. The emitter array also includes individually addressable light emitters that cooperate with readout operations of one or more controllers associated with the detector array.

In relation to wearability, embodiments also relate to a wearable component that interfaces the emitter array and other system components to the head region of the user during use, in order to assess brain activity in a portable manner. The wearable component includes aspects that reliably bias optodes coupled to the emitter array and/or the detector array to the user's head as the user moves about in his or her daily life.

In relation to brain activity sensing and generation of outputs for optical tomography, embodiments also relate to a detector array that can be included with the BCI, where the sensor system enables detection and decoding of brain activity by optical tomography. In some embodiments, detection methodologies other than optical tomography may be used by the detector array. The sensor system includes an array of pixels arranged as grouped pixel units to provide increased dynamic range. One or more of the grouped pixel units can operate in a saturated mode while providing information useful for decoding brain activity. Furthermore, the grouped pixel units are arranged to enable fast readout by a controller (e.g., line scanner or an alternative controller), thereby increasing detection and decoding ability by systems implementing the sensor design. The grouped pixel units of the sensor system are aligned with optical fibers of an interface to a body region of a user, where the optical fibers can be retained in position relative to the grouped pixel units by an optically transparent substrate that provides mechanical support while minimizing factors associated with divergence of light transmitted through optical fibers.

2. System Environment

FIG.1Ais a block diagram of a brain computer interface (BCI) system for detecting and decoding brain activity of a user, according to an embodiment.FIG.1Bis a schematic of an embodiment of the BCI module shown inFIG.1A, according to an embodiment. The system100aincludes a brain computer interface module110, a network130, additional sensors140, a user device150, and at least one computing device160. The BCI module110further comprises an emitter array115, a detector array119, and a controller115. By integrating both of the emitter array115and the detector array110into the BCI module110, the BCI module110is configured to transmit light to a head region of a user and to receive light signals reflected from the head region of the user. The emitter array115and detector array110are coupled to a controller115, which powers both arrays and processes and decodes neural stream signals for delivery of feedback to a user device150through a network130.

As described in relation to other system components above and below, the BCI module110houses the emitter array115, the detector array110, and the controller115. The housing is head-mountable for positioning the signal transmitting emitter array components at a head region. The housing may be composed of a polymer material and/or any other suitable materials. The BCI module110is thus designed to be worn by a user during use. Emitters of the emitter array115and sensors of the detector array110may be positioned and retained at the head region of the user through the interface module110. Accordingly, the interface110is configured to enable characterization of brain activity from one or more regions of the user's brain through a non-invasive method. Specifically, the one or more of emitters of the emitter array115emit light and the one or more sensors of the detector array110capture light signals reflected from the head region of the user based on the emitted light. In some embodiments, the BCI interface module110is designed to fully cover the head of the user. In other embodiments, the BCI interface module is designed to cover a portion of the head, depending on the region(s) of interest associated with application for decoding brain activity.

In various embodiments, the emitter array115and detector array110enable optical tomography methods for receiving neural signals from the user, where the signals can be subsequently decoded and used for other applications (e.g., as control inputs that allow the user to control behavior of devices in his or her environment). Emitters of the emitter array115emit a signal that is absorbed and/or attenuated by neurons or networks of neurons in the region of the brain, and/or cause a physiological response that can be measured. Sensors of the detector array110detect a signal (e.g., backscattered light) from the same region of the brain. In one embodiment, the signal emitted by the emitters and captured by the sensors is light in the visible spectrum. Additionally, or alternatively, the signal emitted by the emitters of the emitter array115and captured by the sensors of the detector array110may be light in the non-visible spectrum.

The emitter array115communications with the controller115to receive instructions that enables the emitter array115to transmit light. The controller115may also function as a computing component of the system electronics. For example, the emitter array115receives inputs from the controller115and provides inputs to a particular emitter of the emitter array115to coordinate light emission from the emitter array115(e.g., in relation to operation of the detector array110in coordination with the emitter array115). More specifically, the emitter array115receives instructions for transitioning the emitters between operation states (e.g., on states, off states) and/or within variations of operation states (e.g., a high power mode in the on state, a low power mode in the on state, etc.). The emitter array115and individual emitters are described in more detail below.

The detector array110detects the signals emitted by the emitter array115through the coupling of the sensors in the BCI interface module110to the user. The detector array110may also be in communication with the controller115to enable sensing, signal pre-processing, and/or signal transmission functions of the detector array110. The detector array110may also be in communication with the controller115for computing functionalities, for example to support detection operation modes (e.g., sensor scanning modes) and/or other operation modes (e.g., signal transmission modes) of the detector array110. In relation to sensors of the detector array110, the sensors may include complementary metal oxide semiconductor (CMOS) architecture and/or another architecture, as described in more detail below.

The controller115encodes the reflected optical signals detected by the detector array110into signals which may be processed by one or more controllers associated with one or more of the computing device160and the user device150.

The system100can additionally include other sensors140for detecting user behavior and/or other biometric signals that can supplement data processed by the BCI module110. In some embodiments, the additional sensors140are also connected to the controller115, which acts as a power component and/or a computing component to enable the additional sensors140to provide signals useful for decoding brain activity of the user, as described in more detail below.

In addition to the computing capabilities of the controller115, the system100acan additionally, or alternatively, implement other computing device160and/or a user device150to aid in particular computations or processing steps, for instance, through the network130. Examples of computing devices160and/or user devices150include a personal computer (PC), a desktop computer, a laptop computer, a notebook, a tablet PC executing an operating system, for example, a Microsoft Windows-compatible operating system (OS), Apple OS X, and/or a Linux distribution. In other embodiments, the computing devices and/or user devices can be any device having computer functionality, such as a personal digital assistant (PDA), mobile telephone, smartphone, wearable computing device, or any other suitable computing device. The computing component160and/or other computing devices can execute instructions (e.g., computer code) stored on a computer-readable storage medium in order to perform the steps and processes described herein for enabling unspoken communications for control of other systems by a user. Collectively, the controller115and any other computing devices160, with the network130, can operate as a computing system for implementation of methods according to specific applications of use of the system100.

In some embodiments, the computing devices160or the user device150, or a controller (not shown) connected to the computing devices160or the user device150, determines intentions of the user from signals provided by the detector array110, where the intentions describe user wishes in relation to interacting with electronic content or a virtual assistant. In one embodiment, a computing device160determines the intentions that correspond to the neural signals that were gathered by the detector array110by applying a predictive model that is trained to predict intentions from neural activity. The computing device160may train the predictive model using training data including gathered experimental datasets corresponding to neural activity of previously observed individuals. Intentions can be decoded into communication related components (e.g., phonemes, words, phrases, sentences, etc.). In some related embodiments, computing device160enables a user to access an online social networking system, and therefore, allows users to communicate with one another through the online social networking system. As such, the computing device160may communicate on behalf of the individual through the network130with other computing devices (e.g., computing device160, user device150) of the social networking system. In some embodiments, the computing device160can communicate on behalf of the individual to other computing devices using the predicted phonemes, words, phrases, and/or sentences. In alternate embodiments, functionality described above with reference to the computing device160may be performed by a controller connected to a user device150.

In addition to the configuration illustrated inFIG.1B, multiple identical BCI modules110may be simultaneously placed on the head of a user with synchronous functionality. In another embodiment, multiple BCI modules110may be interlocked to operate in conjunction with various headset and head-mounted display designs.

The network130facilitates communications between the one or more computing devices. The network130may be any wired or wireless local area network (LAN) and/or wide area network (WAN), such as an intranet, an extranet, or the Internet. In various embodiments, the network130uses standard communication technologies and/or protocols. Examples of technologies used by the network130include Ethernet, 802.11, 3G, 4G, 802.16, or any other suitable communication technology. The network130may use wireless, wired, or a combination of wireless and wired communication technologies. Examples of protocols used by the network130include transmission control protocol/Internet protocol (TCP/IP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (TCP), or any other suitable communication protocol.

The system100described above operates to enable optical tomography or optical topography-associated modalities for decoding neural activity. The system100can characterize blood/tissue characteristics of the user through diffuse optical tomography/topography (DOT) modalities, in relation to characterizing cerebral blood flow, cerebral blood oxygenation, and/or other features indicative of brain activity. The system100can additionally or alternatively support other optical tomography or near-infrared spectroscopy approaches, including one or more of: functional near-infrared spectroscopy (fNIRS), functional time-domain near-infrared spectroscopy (TD-fNIRS), diffuse correlation spectroscopy (DCS), speckle contrast optical tomography (SCOT), time-domain interferometric near-infrared spectroscopy (TD-iNIRS), hyperspectral imaging, polarization-sensitive speckle tomography (PSST), spectral decorrelation, auto-fluorescence tomography, and photoacoustic imaging.

4. System Components

4.1 General System Architecture

FIG.2Ais an isometric view of components of the BCI module shown inFIG.1Baccording to an embodiment. The BCI module200includes optical components, which in combination form an emitter array115and a detector array110, that are held between the clamp205and the heat sink enclosure245. Electrical components, which in combination form a controller115, are positioned between the heat sink enclosure245and the rear heatsink enclosure255. This configuration allows for heat dissipation of the optical components to be independent from the heat dissipation of the controller115. In other embodiments, the optical components and the controller115share the same heatsink and are housed together within the same housing.

The emitter array115includes a plurality of emitters and an emitter ferrule array210. Each emitter of the emitter array115is configured to emit light towards the head of a user and is encased by an emitter ferrule of the emitter ferrule array210. The emitted light travels through the ferrule onto the head of the user. In some embodiments, the emitter ferrule is a plastic casing. In alternate embodiments, the emitter ferrule may be designed using alternate or additional materials. Additionally, the optical components encased in a ferrule are sealed by a ferrule cap, which couples to an exposed end of the ferrule. Emitter ferrules of the emitter ferrule array210are coupled to emitter ferrule caps215in a one-to-one ratio.

Similarly, the detector array110comprises a plurality of detectors and a detector ferrule array220. Detectors of the detector array110are configured to capture, via sensors, signals from the head region of a user based on light emitted by the emitter array115. Each detector is encased by a detector ferrule of the detector ferrule array220, which directs optical signals from the head of the user to the encased detector. In some embodiments, each detector ferrule of the detector ferrule array is coupled to a detector ferrule cap in a one-to-one ratio.

The BCI module200may additionally include a ferrule retaining membrane225. The retaining membrane225is configured to couple each ferrule of the emitter ferrule array210and the detector ferrule array220and to hold each ferrule of the array in a fixed position. In some embodiments, the ferrule retaining membrane225is made of a stretchable material, for example a rubber material, that provides an elastic force allow both the emitter and detector ferrule arrays210and220to conform to the shape of the user's head. In alternate embodiments, the ferrule retaining membrane225is designed using a different material with similar elastic properties.

The BCI module200further comprises a detector flexible printed circuit assembly230and an emitter flexible printed circuit assembly235. Both flexible printed circuit assembly (FPCA)230and235are designed to be sufficiently flexible, stretchable, or both to conform to the head of a user when the BCI module200is worn by the user. The FPCA230and235are designed with a combination of rigid sections and flexible sections. Ferrules of the emitter ferrule array are coupled to rigid sections of the emitter FPCA235and ferrules of the detector ferrule array are coupled to rigid sections of the detector FPCA230. Each rigid section of the FPCA is connected to at least one flexible section. As will be discussed further with reference toFIGS.7A-E, the configuration of the FPCA with multiple interconnected flexible and rigid sections enables the FPCA to conform to the shape of a user's head. For the sake of illustration, the detector FPCA230is configured with a different arrangement of flexible and rigid sections than the emitter FPCA235. However, it is to be understood that the detector FPCA230and the emitter FPCA235may be designed with the same or a similar arrangement of flexible and rigid sections.

Each rigid section additionally includes electronics for emitting or detecting light. For instance, the electronics mounted on rigid sections of the emitter FPCA235may include an emitter package and control circuitry for emitting light through an emitter ferrule onto the head of a user. Moreover, electronics mounted on rigid sections of the detector FPCA230may include a detector package and control circuit for detecting light reflected from the head of the user. The FPCAs230and235include traces for routing signals between a controller (e.g., controller circuit250) and the electronic components.

In some embodiments, the FPCAs230and235have multiple connectors to connect to a corresponding controller. For example, the detector FPCA230or the emitter FPCA235may have two connectors for connecting to a controller board to simplify or enhance the routing of the traces of the FPCA for connecting the electronic components to the controller.

The FPCA retaining ring240is configured to hold both of the detector FPCA230and the emitter FPCA235in place when the BCI module is worn by a user.

Between the heatsink enclosure245and the rear heatsink enclosure255, the BCI module200includes a controller circuitry250. In the illustrated embodiment, the controller circuitry250has a first circuit board with first connectors to connect to the detector FPCA230and a second circuit board with second connectors to connect to the emitter FPCA235. Each circuit board of the controller circuitry additionally includes board-to-board connectors. In some embodiments, the controller circuitry may include a single circuit board configured to connect to both the detector FPCA230and the emitter FPCA235.

Additional sensors may be mounted to the FPCA235adjacent to emitters of the emitter ferrule array210and detectors of the detector ferrule array220. Examples of the additional sensors record measurements including, but not limited to, temperature measurements, inertial measurements, strain measurements, air pressure measurements, electrical voltage measurements, electrical current measurements, capacitance measurements, and impedance measurements.

The BCI module200is worn on the head of a user, for example as shown inFIG.1B. The BCI module200is worn such that each ferrule of the emitter ferrule array210and the detector ferrule array220are in contact with the user's head. Because of the flexible properties of the FPCA's230and235, the BCI module200can be worn on any part of the user's head to measure any region of the brain.

FIG.2Bis an isometric view of components of an alternative BCI module shown inFIG.1Baccording to an embodiment. Components of the BCI module255may be functionally similar to those of the BCI module200, but may differ in geometry. The BCI module260comprises a clamp265that encloses the skin-facing side of the BCI module260. The clamp265is a design variant of the clamp205with a different geometry and screw count compared to the clamp205. Additionally, the BCI module260includes a ferrule soft collar that provides an additional retention force to supplement the retention force provided by the ferrule retaining membrane225.

As illustrated in the extruded views of the detector array and the emitter array, the emitter ferrule caps and the detector ferrule caps may differ in design and geometry to accommodate the varying designs of control electronics for the emitter or detector array. Compared to the heat sink enclosure245illustrated inFIG.2A, the BCI module255may include two separate components: a lower enclosure275with mounting features to hold the flexible printed circuit assembly in position and a middle enclosure280. Additionally, components of the controller circuitry250in the BCI module260are geometrically different from the same components in the embodiment illustrated inFIG.2A.

As described above, the BCI module200includes an array of emitters, each of which is encased in a ferrule of the emitter ferrule array210. The emitter(s) function to emit light having suitable parameters, where the light interacts with a region of interest (e.g., the head region of the user), and is subsequently transmitted to a detector array (described below) for characterization of the region of interest. As described herein, optical assemblies involving an emitter of an emitter array and/or additional optical components are referred to as emitter assemblies. Each emitter ferrule encases an emitter package. In some embodiments, each emitter ferrule additionally encases a controller or a portion of the controller configured to operate the emitter package.

The emitter package includes laser light emission elements but can include light emitting diode (LED) elements or other types of light emitters in alternative embodiments. In relation to laser light emission elements, the emitter package can include vertical cavity surface emitting laser (VCSEL) elements with semiconductor architecture for perpendicular beam emission. Use of VCSEL elements contributes to a compact light emission configuration that provides suitable power for wearable applications requiring high power light output for optical detection of characteristics of a multidimensional region of interest (e.g., a head region of a user). In relation to laser light emission elements, the emitter array can alternatively include emitters with conventional semiconductor architecture for edge beam emission from surfaces cleaved from a semiconductor wafer.

In some embodiments, emitters are configured to emit light in the visible spectrum. In other embodiments, emitters are configured to emit light in the non-visible spectrum. For BCI applications involving characterization of brain activity, the emitters include emitters configured to emit red wavelength light and near-infrared light. However, in alternative variations, the emitters are configured to emit only a single wavelength of light, or other wavelengths of light.

Each emitter of the emitter array has its own die physically coupled to an electronics substrate (e.g., emitter FPCA235) of system electronics, such that the emitter is individually addressable in a compact format. In some embodiments, each emitter is packaged into an individual integrated circuit package or chip and mounted on the substrate to be interconnected with other components of the BCI module200. In relation to addressability, each emitter is transitionable between an activated state for emitting light (with different output settings) and a deactivated state. However, in alternative variations, multiple emitters can be associated with a single die physically coupled to an electronics substrate223to enable addressability of emitters in groups, or multiple emitters of the emitter array210can be associated with a single die physically coupled to the electronics substrate223. In alternative embodiments where each emitter does not have its own die, the emitters can, however, still be individually addressable using other wiring architecture.

The emitters may operate in a continuous emission mode for continuous emission of light. The emitters may also operate in a pulsed mode, where periods of light emission are interspersed with periods of non-emission at a desired frequency. Pulses can thus be associated with one or more of: a pulse profile having width characteristics and other pulse shape aspects (e.g., peaks, troughs, etc.); power draw (e.g., in relation to power amplitude); temporal features (e.g., periodicity, frequency of pulses, etc.); and any other suitable pulse features. For example, the emitters operate in a pulsed mode with pulse width modulation (PWM) having a power draw of 27% efficiency at 100 mW of power output.

In relation to non-continuous transmission modes, light emission by emitters may be coordinated with detection by the detector array, where emission of light (e.g., by a subset of emitters) is timed with light detection in phases. In one example, as shown inFIG.3, the emitters of the emitter array emit a first light profile (e.g., a pulse of red light provided within a first time window), followed by a second light profile (e.g., a pulse of infrared light provided within a second time window), followed by a third light profile (e.g., dark period provided within a third time window), where each pulse and dark period is detected sequentially by an embodiment of the detector array described below. As such, scanning by the detector array to generate and decode light signal data can be carefully timed with operation of the emitter array.

Emitters may further be configured to transmit light through optical elements that manipulate light along a path of transmission to a surface of the region of interest. Optical elements can include one or more of: filters, lenses, mirrors, collimation elements, waveguides, other beam shaping elements, and any other suitable optics.

FIGS.4A-Cillustrate example configurations of emitter array115and detector array110in a BCI module, according to an embodiment.FIG.4Adepicts a head-mountable interface according to an embodiment, where the interface module410functions to interface light emitters of the emitter array115(such as an embodiment of the emitter array described above) to the head region of the user during use, and to interface the head region with a detector array (described in more detail below), in order to assess brain activity in a portable manner. The interface module410is configured to be worn by a user and includes an array of ferrules415. The interface module410is secured to the head of the user (or another body region of the user, in relation to signal detection from other body regions) by a retainer420.

The emitters of the emitter array are arranged in a 2D array. The 2D array may be a square array, where the square array can have equal numbers of emitters along its width and height. The size of the array of emitters, in terms of number of emitters, distribution of emitters in space, and spacing between emitters, can be configured based on size of each individual emitter (in relation to size constraints of the wearable system). In alternative embodiments, however, the emitters are arranged in a polygonal array, ellipsoidal array, or in any other suitable manner (e.g., an amorphous array).FIG.4Aillustrates an exemplary configuration440of an emitter array and an exemplary configuration430of a detector array, according to an embodiment. In the illustrated embodiment ofFIG.4A, the emitter array configuration440and the detector array configuration430are separated into distinct sections of the interface module410. However, in alternate embodiments, the emitter array configuration and the detector array configuration may overlap such that emitters and detectors are positioned in the same region of the interface module410. Additionally, the configuration430of the detector array mirrors the configuration420of the emitter array. However, in alternate embodiments, the configuration430may be different from the configuration420.

In most embodiments, the emitter array and detector array are designed with dual wavelength functionality, for example to emit and detect red and infrared light. Because red and infrared light scatters in different ways, a controller in a BCI module may generate a positive image and a negative image, which would allow the BCI system to verify a signal reading with different views. In the embodiments ofFIGS.4B and4C, the emitter array and detector array are arranged in a hexagonal pattern with a mechanical symmetry. Within each array, optical components (i.e., emitters or detectors) are arranged with a uniform spacing, orientation, and distance. Accordingly,FIG.4Billustrates an implementation in which a first subset of emitters of the emitter array are instructed to emit infrared and a second subset of the emitters are instructed to emit red light. Each detector of the detector array is surrounded by six emitters: three emitters configured to emit red light and three emitters configured to emit infrared light.FIG.4Cillustrates an alternate implementation in which each emitter of the emitter array is configured to emit a combination of red and infrared light. The configuration ofFIG.4Callows a single emitter to emit both infrared and red light, which in turn enables the number of detectors on the BCI module to be maximized.

4.3 Integrated Ferrule Assembly

As described herein, an integrated ferrule assembly comprises a ferrule, which is mounted to an FPCA that is configured to conform a head of a user. The ferrule encases electronics that includes, but is not limited to, an optical element (e.g., a sensor of a detector array or an emitter of an emitter array) and a controller (i.e., control electronics) configured to operate the optical element. In implementations in which the optical element is an emitter, the controller generates instructions for the emitter to emit infrared, red, or both light towards the head of the user. In such implementations, the ferrule channels light from the emitter towards the head. Alternatively, when the optical element is a sensor, the controller provides instructions for the sensor to detect light reflected off the head and the ferrule channels the light reflected off the head of the user to the detector.

As described herein, optical packages involving an emitter of the emitter array115and/or additional optical components are referred to as emitter packages.FIG.5Ais a cross-sectional view of an integrated emitter assembly, according to an embodiment. The emitter assembly is configured to emit light towards the head of the user. The integrated emitter assembly500includes a rigid section530of an FPCA. Additionally, an emitter package540and a controller550are attached to the rigid section530of the FPCA. The rigid section530of the FPCA then connects the emitter package540and the controller550to each other. In some embodiments, the emitter package540and the controller550are attached to different sides of the rigid section530of the FPCA. The emitter package540is mounted to a side of the rigid section530of the FPCA such that, when worn by a user, the emitter package540emits light towards the head of the user. The emitter package540includes an optical element, for example a vertical cavity surface emitting laser (VCSEL), a single wavelength LED, a dual wavelength LED, an alternate suitable emitter component or a combination thereof, configured to emit light towards the head of a user. Moreover, as described above, the controller550generates instructions for the emitter package to emit light.

The integrated emitter assembly500further includes a ferrule520which encases the emitter package540. In some embodiments, the ferrule520encloses additional components mounted on the same side of the rigid section of the FPCA as the emitter package540. Although not shown, the integrated emitter assembly500may further include a ferrule cap to seal the emitter package540and controller550in position within the ferrule520and to secure the ferrule520against the rigid section530of the FPCA. The ferrule520is attached to a stretchable ferrule retaining membrane510, which is configured to stretch and flex to conform to the head of a user. When a BCI module is worn by a user, the stretchable ferrule retaining membrane510secures individual ferrules of each of the integrated emitter assemblies500in a fixed position against the head of the user.

An integrated emitter assembly500ais physically and electrically coupled to at least a neighboring integrated emitter assembly500bby a flexible section535of the FPCA. In some embodiments, an integrated emitter assembly500ais coupled to multiple neighboring integrated emitter ferrule assemblies500through multiple flexible sections540of the FPCA. The flexible section535of the FPCA allows each of the integrated emitter assemblies500to move independently from each other, while maintaining physical or electrical connection with each other. As such, the flexible section535of the FPCA allows for electrical signals to be routed to each of the individual integrated emitter assemblies500while allowing the integrated emitter assemblies500to move independently to conform to the shape of a user's head.

As described herein, optical packages involving a sensor of the detector array110and/or additional optical components are referred to as detector packages.FIG.5Bis a cross-sectional view of an integrated detector assembly, according to an embodiment. In comparison to the integrated emitter assembly500, the integrated detector assembly560is configured to direct light reflected off the head of a user towards a detector package. The integrated detector assembly560includes a rigid section590of an FPCA. Additionally, a detector package580and a controller590are attached to opposite sides of the rigid section590of the FPCA. The rigid section590of the FPCA then connects the detector package575and the controller580to each other. In some embodiments, the detector package575and the controller580are attached to different sides of the rigid section590of the FPCA. The detector package575is mounted to a side of the rigid section590of the FPCA such that, when worn by a user, the detector package575is pointed towards the head of the user. The detector package575includes an optical element, for example a silicon photomultiplier, an alternate suitable optical sensor, or a combination thereof, configured to detect an optical signal. As described herein, a silicon photomultiplier is a light sensor based on semiconductor technology that is capable of detecting very small light intensity at a photon-level. The silicon photomultiplier may additionally consist of microcells, each of which is charged with high voltage. When a microcell is triggered by detection of a photon, a predetermined discharge current is generated and flows to a resistor to generate the voltage output. In some embodiments, the amplitude of the voltage output is proportional to the number of triggered microcells. Moreover, as described above, the controller580generates instructions for the detector package575to start detecting light, receives optical signals detected by the detector package575, and encodes the detected optical signal to be sent to a computing device for analysis.

The integrated detector assembly560further includes a ferrule570which encases the detector package575. In some embodiments, the ferrule570encloses additional components mounted on the same side of the rigid section of the FPCA as the detector package575. Although not shown, the integrated detector assembly560may further include a ferrule cap to seal the detector package575and the controller580in position within the ferrule570and to secure the ferrule570against the rigid section590of the FPCA. The ferrule520is attached to a stretchable ferrule retaining membrane565. When a BCI module is worn by a user, the stretchable ferrule retaining membrane565secures individual ferrules of each of the integrated detector assemblies560in a fixed position against the head of the user.

An integrated detector assembly560ais physically and electrically coupled to at least a neighboring integrated detector assembly570bby a flexible section595of the FPCA. In some embodiments, an integrated detector assembly560ais coupled to multiple neighboring integrated detector ferrule assemblies570through multiple flexible sections595of the FPCA. The flexible section595of the FPCA allows each of the integrated detector assemblies560to move independently from each other, while maintaining physical or electrical connection with each other. As such, the flexible section595of the FPCA allows for electrical signals to be routed to each of the individual integrated detector assemblies560while allowing the integrated detector assemblies560to move independently to conform to the shape of a user's head.

FIG.6Ais a side view schematic of a ferrule, according to one embodiment.FIG.6Ashows a ferrule600that can be retained by the stretchable ferrule retaining membrane510(or the stretchable ferrule retaining membrane565). The ferrule600has a first region605and a second region615that are joined at a mid-region610, where the width of the ferrule600is reduced at a mid-region610such that the ferrule600can interlock with a port. in some embodiments, the ferrule600has a varying width along the length of the body625. The width of the ferrule600is smallest at the end of the second region615such that the end of the second region615can comfortably interact with a head region of a user. In alternative embodiments, the ferrule600can have a constant width along the length of the body625and be coupled to the cap in another manner, as described above. The ferrule600can also have wider or smaller end regions, depending on the design considerations of the system (e.g., manufacturability, size, material, etc.).

FIG.6Bis a cross-sectional view of the ferrule illustrated inFIG.6A, according to one embodiment. The illustrated ferrule includes a body625and a cavity630. The cavity630terminates within the second region615of the body625. Alternatively, the cavity630can extend through the first region605and the second region615of the ferrule600. The ferrule600can include a cover620coupled to the body625and/or other components within the cavity630. The cover620can function to seal the cavity630from the external environment in order to protect internal components. The cover620can also function as the interface between the user and the ferrule600. The cover620can be a separate component coupled to (e.g., by an adhesive, interlocking mechanism, etc.) or ensheathing the body625. Alternatively, the cover620can be a continuous piece of the body625.

The cover620can also be made of a material that provides light manipulation functions. For instance, the cover620can be composed of an optically transparent material that allows light transmission without significant loss. In another embodiment, the cover620can be made an optically translucent material to facilitate diffusion of stimulation light. In another embodiment, one or more regions of the cover620can include lenses that affect light transmission through or into the cover. The ferrule600may be designed using optical grade acrylic or polycarbonate, or any other suitable material. Additionally, in some embodiments, a physical vapor deposition coating is applied to the ferrule600to prevent light from leaking from the ferrule600.

The cavity630is configured to house an optical package, for example an emitter package of an integrated ferrule assembly or a detector package of an integrated detector assembly. The cavity630is designed with a volume that allows the ferrule to be placed on top of the entirety of the optical package. Depending on the design of the optical package, the geometry of the cavity630may vary to accommodate differently shaped elements of the optical package, for example sensors or detectors of varying sizes and shapes. Additionally, the cavity630may be designed to channel light toward or away from the optical package without obstructing or affecting the intensity of the channeled light.

In morphology, the ferrule600may be held in position by a ferrule retaining membrane, for example the ferrule retaining membrane225. In some embodiments, the body625has a recessed ring about its external perimeter that forms a portion of an interlocking mechanism such that it can mate with a port of an array of ports in the ferrule retaining membrane, for instance, where the port includes a protrusion about one or more portions of its internal perimeter, the protrusion operable to lock with the recess of the body625. The width of a middle portion of the body625may be smaller than one or both end regions of the body625such that the ferrule600is retained in position without an adhesive or other attachment mechanism. In an alternative embodiment, the body625of a ferrule600has a constant width. In still other embodiments, the body625may be cylindrical, polygonal, or any other suitable shape for encasing control electronics and directing light.

The ferrule635may be made of a single material or a composite material with physical properties capable of supporting the optical fiber642. The material can have mechanical properties (e.g., ductility, strength) suited support the optical fiber642. In relation to mechanical properties, the material(s) of the ferrule635can have a compressive strength, a shear strength, a tensile strength, a strength in bending, an elastic modulus, a hardness, a derivative of the above mechanical properties and/or other properties that enable the ferrule635to move with respect to the stretchable ferrule retaining membrane510while maintaining its position relative to other ferrules of a BCI module. The ferrule635may be made of polycarbonate; however, the body can be made of another material (e.g., polymeric material, non-polymeric material).

In relation to electrical properties, the material(s) of the ferrule635can have a conductivity, resistivity, a derivative of the above electrical properties and/or other properties that support signal transmission through the embedded optical fiber645. The ferrule635may be made of an insulative material in order to prevent excess noise from propagating from the emitter package to the user and/or to other components of the system.

In relation to optical properties, the material(s) of the ferrule635can have optic properties that enable signal transmission through the embedded optical fiber645. The ferrule655can include an optically opaque adhesive configured to facilitate signal transmission between the first region605and the second region615of the ferrule600. The body625of the ferrule can also be composed of an optically opaque adhesive. In alternative embodiments, the ferrule600can be composed of any suitable material.

FIGS.6C-6Eillustrate alternate ferrule designs.FIG.6Cillustrates a ferrule design635in which an optical fiber642is embedded into the ferrule635to guide emitted and reflected light to and from the optical package640. The optical fiber642may vary in length depending on the design of the ferrule. The illustrated embodiment ofFIG.6Cmay be implemented to direct light from the head of a user towards a detector assembly or from the emitter assembly to the head of the user. The body of the ferrule may be cylindrical, polygonal, or any other suitable shape for supporting the embedded optical fiber642.

FIG.6Dillustrates an alternate ferrule design650in which the ferrule geometry is designed such that the light is focused when emitted from the optical package640(e.g., an emitter assembly) towards the head of a user or when detected by the optical package640(e.g., a detector assembly).FIG.6Eillustrates an alternate ferrule design655in which a stack of lenses, including first lens657and second lens659, are installed within the ferrule body to focus the light emitted by the optical package640or detected by the optical package640. Each of the first lens657and the second lens659may be integrated in the plastic casing of the ferrule to optimize an amount of light output at the tip of the ferrule. In some embodiments, a glass ball lens may be installed within the ferrule body.FIG.6Fillustrates a variation of the ferrule design655(illustrated inFIG.6E) in which first lens is encased in a first lens housing662and the second lens is encased in a second lens housing663.FIG.6Gillustrates an alternate ferrule design670in which a single ball lens672is used to direct light towards or away from the optical package640. The ferrule design670may be used to reduce the overall form factor of an individual ferrule.

FIG.6Hillustrates an alternate ferrule design680involving a light pipe685. Based on a change in index of refraction between the ferrule680and the surrounding air, the ferrule680guides light emitted by the optical assembly640towards the head of a user. The illustrated embodiment ofFIG.6Hmay be implemented in an emitter package to emit light towards the head of a user.

A ferrule, for example the ferrule designs illustrated inFIGS.6C to6H, may additionally be coated or metalized using a vapor coating a painting to prevent light leakage into or out of the ferrule.

Additionally, a ferrule of either an integrated detector assembly or an integrated emitter assembly may be designed to carry an electrical current. For example, the ferrule may be made using an electrically conductive material or may be coated with an electrically conductive material. As described herein, such an electrically conductive ferrule may be referred to as an optoelectrode. When worn by a user, a tip of the ferrule body is placed in contact with the skin of a user, for example the scalp of a user, and the opposite end of the ferrule is positioned in contact with an FPCA. In addition to directing optical signals towards a detector assembly, an electrically conductive ferrule, also carries electrical currents detected at the tip of the ferrule to circuitry on the FPCA, which measures electrical properties of the scalp. The measured properties include, but are not limited to, capacitance, impedance, or voltage at the location where the ferrule contacts skin of the user.

Each optoelectrode includes an optical interface at the tip of the optoelectrode for allowing an optical signal to travel from the scalp of a user into the ferrule. The optical signal then is able to travel through the ferrule (i.e., the core of the optoelectrode) to an optical package. Alternatively, the optical interface at the tip of the optoelectrode allows for a signal through the ferrule into the scalp of the user.

Furthermore, each optoelectrode includes an electrical interface at the tip of the optoelectrode to transfer electrical signals to or from the scalp of the user through the electrically conductive outer surface of the optoelectrode.

In some embodiments, such an optoelectrode may be implemented in a BCI module to measure electrical biological signals to aid in the decoding of optical biological signals, for example electroencephalography (EEG) signals. In other embodiments, optoelectrodes may be used to confirm ferrule contact with skin of a user and to confirm a quality optical transmission for both the emitter array and the detector array.

FIG.6Jillustrates an embodiment of an optoelectrode involving the ferrule design655described with reference toFIG.6E. The ferrule is covered with an electrically conductive coating698the enable an electrical continuity between the scalp of a user and contact circuitry697on the FPCA. The contact circuitry697decodes and processes the electrical signal to measure the electrical properties described above. Similarly,FIG.6Killustrates an embodiment of an optoelectrode involving the ferrule design680described with reference toFIG.6H. The illustrated ferrule is covered in an electrically conductive coating698to communicate electrical signals to the contact circuitry697for processing. In addition to its conductive properties, the coating698may also function to block out light or prevent light leakage.

FIG.6Iillustrates an alternate ferrule design690, which enables the ferrule to make greater contact with the skin than the previously discussed designs. The ferrule design690includes a ferrule spring695. When the ferrule is not in contact with the skin (e.g., the scalp of a user) the spring is extended and ferrule design690is in a resting state. However, contact with the user's scalp results in a force applied to the tip of the ferrule and the compression of the ferrule spring695. Compression of the ferrule spring692triggers a controller (not shown) to generate instructions for the optical assembly to emit light towards the user's head. In some embodiments, the compressive force applied to the ferrule spring695is provided by retainer420that is used to secure the BCI module410to the head of the user.

In some embodiments, the ferrule design690includes a cavity that houses the ferrule spring695. Moreover, the cavity may house the optical package640. The optical package640may be above the optical package640. In some embodiments, the optical package640may be connected to the FPCA via a flexible cable or ribbon.

4.4 Flexible Printed Circuit Assembly

For a BCI module to accurately and most effectively detect signals, for example signals that characterize brain activity, the BCI module should conform to the geometry and structure of the heads of each individual user. When the BCI module is able to conform in such a way, the performance of each emitter ferrule assembly and detector ferrule assembly is increased. Accordingly, the BCI module comprises a flexible printed circuit assembly (FPCA) in combination with a stretchable ferrule retaining membrane to interconnect at least one emitter assembly and one detector assembly of the BCI module, or to connect the emitter assemblies and the detector assemblies to the controller circuitry250.

To enable the BCI module to conform to a head of a user, each ferrule assembly is capable of moving independently to be able to conform to the shape of the head of the user. To enable the flexibility of the ferrule array, individual ferrules of the ferrule array are interconnected by a stretchable ferrule retaining membrane.FIG.7Ais a side view of a stretchable ferrule retaining membrane configured to hold a plurality of ferrule assemblies in position, according to one embodiment. The stretchable ferrule retaining membrane700is held in position by an enclosure710, for example lower enclosure275or heatsink enclosure245, on one face of the membrane700and a clamp725on the opposite face of the membrane700. Between the membrane enclosure and the membrane retainer, the stretchable ferrule retaining membrane700retains a plurality of ferrules in position. The stretchable ferrule retaining membrane700enables conformity of the plurality of ferrules, both individually and collectively, to various head curvatures. When a force is applied to either end of the stretchable ferrule retaining membrane700or at any point along the membrane, each ferrule of the ferrule array conforms to the unique curvature of the head of the user. In some embodiments, the membrane700is designed out of poured or compressed molded rubber or silicone. In alternate embodiments, the membrane700may be designed using alternate materials or techniques that similarly enable the flexibility and conformity of the plurality of ferrules.

Just as the stretchable ferrule retaining membrane700enables the conformity of each ferrule of a ferrule array to conform to unique curvatures, a BCI module comprises an FPCA is configured to enable the electrical connectivity of the electronic components of the BCI module (e.g., the emitter and detector packages and their respective controller) while also allowing flexibility of the electronic components. Electric components mounted to the FPCA are able to flex due to the stretchable properties of the FPCA, which are enabled by the geometries of flexible sections of the FPCA.FIG.7Billustrates an FPCA with flexible sections connected to rigid sections, according to an embodiment. The FPCA730enables mechanical flexibility such that the optical packages and control components can conform to the curvature of a user's head. The FPCA730includes rigid sections740a,740b, and740c. As described with reference toFIGS.5A and5B, an optical package (e.g., an emitter package or a detector package) and a controller are mounted on opposite sides of each rigid section740. Each rigid section additionally has a ferrule mounted on the side of the optical package and a ferrule cap on the opposite side, forming an integrated ferrule assembly (i.e., the integrated emitter assembly500and integrated detector assembly570). In alternate embodiments, both the controller and the optical assembly may be mounted to the same side of a rigid section740.

The FPCA730enables the flexibility and conformity of the BCI module. However, once the FPCA has conformed to a user's head, each rigid section740independently holds the mounted ferrule assemblies (e.g., ferrule, optical package, and control components) in a fixed position, such that each ferrule assembly remains in continuous contact with the head of the user.

In one embodiment, emitter assemblies and detector assemblies are mounted to rigid sections740of the FPCA in a hexagonal formation. In other embodiments, the emitter and detector assemblies may be mounted to rigid sections in any other suitable geometric formation. Additionally, consistent with the description ofFIGS.4A and4B, optical assemblies are mounted to rigid sections such that each rigid section with detector assemblies are surrounded by rigid sections with emitter assemblies. For example, if a detector assembly were mounted to rigid section740b, emitter assemblies would be mounted to each of rigid section740aand740c.

In some embodiments, the rigid sections are made from a flexible, stretchable substrate and a rigid substrate mounted on the flexible substrate. For example, the flexible substrate may be a plastic or other polymer having conductive strips or traces and contact pads printed (or etched) onto it. The rigid substrate is then electrically connected to the flexible substrate to allow electrical signals to be transferred from the flexible substrate to the rigid substrate. In some embodiments, one or more components of the integrated ferrule assembly (e.g., an optical package or a control component) is mounted onto the rigid substrate. Moreover, in some embodiments, one or more components of the integrated ferrule assembly is mounted to the flexible substrate. For example, the control components may be mounted to the rigid substrate and the optical package may be mounted to the flexible substrate.

Each of the rigid sections745is interconnected to at least one neighboring rigid section by one or more flexible sections745. In the illustrated embodiment ofFIG.7B, rigid sections740aand740bare interconnected by flexible section745aand rigid sections740band740care interconnected by flexible section745b. As described above with reference toFIGS.5A and5B, the flexible sections745enable each ferrule to flex and conform to the curvature of a user's head independent of other ferrules of the ferrule array.

The flexible sections745of the FPCA has a curved shape. In some embodiments, each flexible section745has an S shape with rigid sections740attached to each end of the S shaped flexible section. Although the curved shape of the flexible section745increases the length a trace travels to cross from a first rigid section to a second rigid section, the curved shape allows for increased ability of the FPCA to stretch and flexibility of electric components mounted to the FPCA. In addition to the illustrated and described designs of the flexible sections745, any alternate design of flexible sections may be used to improve or enable the flexible sections, or the FPCA730as a whole, to stretch to conform to a head of a user. As such, the curved shape of the flexible sections745allows for a position of the ferrule assemblies to deviate from each other while maintaining physical and electrical connectivity between them.

FIG.7Cillustrates an alternate embodiment of an FPCA with flexible sections connected to rigid sections, according to an embodiment. The illustrated FPCA750includes rigid sections740for mounting emitters, detectors, control circuits, and additional sensors. In the illustrated embodiment ofFIG.7C, each rigid section of the FPCA may be designed with unique geometries depending on the circuit elements mounted on the rigid section. For example, rigid section740dand rigid section740eare designed with unique geometries to mount different optical elements or circuit components. The flexible sections of the FPCA are designed with geometries that enable each rigid section to stretch and flex relative to any neighboring rigid sections. Flexible sections, for example flexible sections745c,745d,745e, and745f, may be designed unique geometries to enable each rigid section to stretch and flex to various degrees or orientations. Additionally, each flexible section may be designed as part of a flexible branch in a set of multiple interwoven flexible branches that lead back to a common printed circuit board. The rigid sections740may then be mounted on specific sections of the stretchable branches.FIG.7Dillustrates isometric views of the FPCA750, according to an embodiment.

In some embodiments, the flexible branches of the FPCA are formed by cutting a flexible printed circuit board (PCB). Each of the flexible branches may then be folded to a desired position. In some embodiments, the FPCA includes multiple flexible PCBs that are interwoven together. For example, the embodiment ofFIG.7Dincludes four flexible PCBs, two flexible PCBs to form flexible sections for detectors and two flexible PCBs to form flexible sections for emitters. Each flexible PCB may include a connector760at one end to allow the FPCA to connect to a controller board that controls the operation of the emitters and detectors mounted on the FPCA. The flexible branches of the FPCA may be designed stretch and flex as described above with reference to the flexible sections745.

Each of the flexible branches include metal traces for routing an electrical signal from the connector760to a rigid section740mounted on the flexible branch. In some embodiments, each branch includes a separate metal trace (or set of metal traces) for each of the rigid sections mounted on the flexible branch. In other embodiments, each flexible branch includes a single set of metal traces that are shared be every (or a subset) of rigid sections mounted on the flexible branch. For example, the set of wires may be a bus that can be multiplexed to allow signals to be addressed to individual rigid sections.

Additionally, in some embodiments, each flexible branch has a corresponding set of terminals in the connector760. As such, a controller circuit connected to the connector760is able to send control signals and receive measurement signals from each of the stretchable branches independently. In other embodiments, each flexible branch is coupled to one or more buses. In this embodiment, the signals corresponding to each individual stretchable branch may be multiplexed to allow a controller circuit to communicate with emitters or detectors mounted on each flexible branch independently.

FIG.7Eis an isometric view of the FPCA, ferrule array, ferrule caps, and stretchable ferrule retaining membrane, according to an embodiment. The stretchable ferrule retaining membrane755retains each ferrule of the ferrule array765in a fixed position. As illustrated, the membrane755includes a plurality of retainer units760, each of which can hold a single ferrule of the ferrule array765. The FPCA750is overlaid onto the stretchable ferrule retaining membrane755, such that rigid sections of the FPCA750align with at least a subset of retainer units of the stretchable ferrule retaining membrane755. Once aligned and overlaid, optical packages mounted to each rigid section are inserted into a cavity of a ferrule retained by the corresponding retainer unit. The ferrules are secured to the FPCA750by ferrule caps of ferrule cap array770, such that a rigid section of the FPCA together with the optical package and controller are encased between a ferrule and a ferrule cap.

4.5 Signal Generation Methods Using Emitter and Detector Arrays

FIG.8is a flow chart illustrating a method for generating and processing optical signals, according to an embodiment. The method800functions to enable decoding of optical signal-derived data from a region of interest, for example using high dynamic range sensors of a detector array and individually addressable light emitting components of an emitter array in a compact BCI module.

As shown inFIG.8, an emitter array, through individual emitter assemblies, emits810light onto a multidimensional region of interest, which functions to provide controlled transmission of light that can be received by a sensor for characterization of the region of interest. The light transmitted from the multidimensional region of interest can include light in the non-visible spectrum and/or light in the visible spectrum, can include a single wavelength of light or multiple wavelengths of light, can include naturally encoded information (e.g., due to physiologically induced phenomena), and/or can include synthetically encoded information (e.g., due to polarization or other light manipulating optics positioned along a light transmission pathway). The transmitted light can be associated with any energy associated factors (e.g., power, duration of transmission, intensity), waveform factors (e.g., pulsed, non-pulsed, waveform shape), temporal factors (e.g., frequency of signal transmission), and/or any other suitable factors.

In relation to the interface described above, the multidimensional region of interest is a head-region of the user, where noninvasively-acquired light signals can be used to decode brain activity of the user through the head region. The head region can include one or more of: a frontal region, a parietal region, a temporal region, an occipital region, an auricular region, an orbital region, a nasal region, or an infraorbital region. Additionally, or alternatively, the head region can include other cranial or facial regions including one or more of: an oral region, a parotid region, a buccal region, or any other suitable region of the head of the user. In alternative embodiments, the multidimensional region of interest can be associated with another anatomical region of the user. Additionally, or alternatively, the multidimensional region can be associated with a surface or volume of material of another object.

Individual detector assemblies of a detector array detect and capture820light emitted onto the region of interest. The captured light is derived from light that originated at the set of emitter assemblies and interacted with the multidimensional region of interest (e.g., the head of the user). The captured light can thus be associated with light sourced from individually addressable emitter assemblies, where light output from the emitter assemblies can be timed according to scanning of the detector according to methods described below. The detector assemblies can additionally or alternatively capture light derived from ambient light from the environment of the user, where the ambient light has interacted with the multidimensional region of interest. Light detections through the set of detector assembly can, however, come from any other suitable source.

After light is captured820by detector assemblies, a controller associated with the detector array generates830light-derived signals for characterization of the region of interest. Alternatively, the captured light may be received by a controller associated with an individual detector package of the array. The scanning operation(s) of the controller provide fast readout, in order to facilitate rapid processing and decoding of information (e.g., neural stream data) derived from incident light on the illuminated regions of the head.

In generating830light-derived signals, the controller reads electrical signals produced from the reflected light detected by the detector array. The controller reads the detected light reflections sequentially in order to produce fast readout speeds. Furthermore, the controller reads the detected reflections in a linear manner. However, in alternate embodiments, the controller can read detected reflects in any other suitable path. The scanning operation can read full frames of signals of reflected light and/or can read less than full frames of signals (e.g., a central line of signals along the scan path). The scanning operation can be specified with parameters related to speed (e.g., in terms of frame rate), power consumption, or any other suitable parameter.

As described above, the detector array, with the controller, can be configured to coordinate scanning operation with light output through individually addressable light emitters (e.g., light emitters of the VCSEL array, LED light emitters, etc.). As such, to generate830the light-derived signals, a portion (e.g., a first subset) of emitter assemblies are activated and the detector array scans the illuminated region of the head in coordination with activation of the portion of light emitters. Then, a second portion (e.g., a subset the same as or different from the first subset) of emitter assemblies are activated and the detector array scans the illuminated region of the head in coordination with activation of the second portion of light emitters. In this example, portions of emitter assemblies can be activated in a manner to minimize crosstalk/interference due to light emission, or to serve any other suitable purpose. However, the first portion and the second portion of emitter assemblies activated can be associated with targeting different portions of the region of interest (and not necessarily to minimize crosstalk/interference). However, in alternative embodiments, coordination of timing between light emission and scanning by the detector array can be conducted in any other suitable manner.

Generating light-derived signals can facilitate extraction of features from illuminated regions of a user's head, where features can be extracted from illuminated central regions and/or unilluminated edge regions of the grouped pixel units in order to increase dynamic range of the sensor outputs by several orders of magnitude relative to unsaturated sensor configurations. Features related to saturated portions can include positions of boundaries between a saturated central region and unsaturated edge regions (indicative of total power), diameter of a saturated central region, projected area of a saturated central region, or any other suitable shape-related features associated with the saturated central region. Features related to unsaturated edge regions can include positions of boundaries between unsaturated edge regions, slope related features (e.g., rates of decay) of a heel portion of an unsaturated edge region, features related to integrated areas under an intensity curve corresponding to unsaturated edge regions, or any other suitable shape-related features associated with the unsaturated edge region. While features associated with intensity are described above, features that can be derived from the generated light signals can include features of any other suitable light-related parameter. Furthermore, in relation to the region of interest being at a head region of the user, the features of interest can be decoded to distinguish different types of brain activity of the user, which can be used as control inputs for controlling operation states of other systems (e.g., virtual assistants, interactions with an online social network, smart home devices, etc.). As described in more detail in relation toFIGS.10A-10Ebelow.

As shown inFIG.1A, the BCI system can include additional sensors140for detecting user behaviors and/or other biometric signals that can supplemental data.

FIG.9Aillustrates a schematic of a camera subsystem925, according to an embodiment, andFIG.10Billustrates a schematic of the camera subsystem925shown inFIG.9A. As shown inFIG.10, the additional sensors can include one or more cameras950of a camera subsystem925, which function to generate image data of the user and/or of an environment of the user. The cameras925utilize light of the visible spectrum, but can additionally or alternatively include sensors that utilize any other portion of the electromagnetic spectrum (e.g., infrared spectrum). The camera subsystem925can use image sensors of the camera(s)950to capture image data and/or video data. In relation to image data and video data, the camera subsystem925can be configured to capture data with sufficiently high resolution to capture features of interest of the user (e.g., pupil position and orientation, facial features, body movements, etc.) and/or of the environment of the user (e.g., states of objects in the environment of the user), where applications of user tracking and environmental tracking are described in more detail below.

In relation to the wearable interface110described above, the camera(s)950of the camera subsystem925can be coupled to the wearable interface (and electronics subsystem1050) shown inFIGS.10A and10B, in a manner that orients the camera(s) with a field of view capturing the face (or a portion of the face) of the user, and/or with a field of view capturing an environment of the user (e.g., from a point of view of the user). As such, the camera subsystem925can include a first camera960coupled to (e.g., mounted to, electromechanically coupled to, etc.) a portion of the wearable interface and in an inward-facing orientation to provide a field of view capturing the face of the user, in order to generate eye tracking data (e.g., in relation to coordinates of objects the user looks at, in relation to swell time) of the user and/or facial expressions of the user. The camera subsystem925can also include a second camera970coupled to (e.g., mounted to, electromechanically coupled to, etc.) a portion of the wearable interface and in an outward-facing orientation to provide a field of view capturing the environment of the user, in order to generate image data of objects or environments with which the user is interacting. The camera subsystem925can, however, have more than two cameras coupled to the wearable interface or other portions of the system in another orientation. Additionally, or alternatively, the camera(s) can be fixed in position, or can be actuated to adjust field of view.

As indicated above, the camera subsystem925can cooperate with other portions of the system described above, in applications where capturing interactions of the user with the environment of the user can be combined with decoded brain activity of the user in a useful manner. In one such application, the system can monitor, by way of cameras950of the camera subsystem925, objects that the user is interacting with in his/her environment by generating and analyzing images of eye motion of the user, head motion of the user, gaze of the user, and/or line-of-sight to objects in the user's environment, decode an intention of the user from brain activity of the user acquired through the detector array described above, and apply the intention as an input to control an operational state of the object. Examples of objects can include electronic content provided at a display (e.g., of a computer, of a wearable device, of an artificial reality system, of a virtual reality system, of an augmented reality system, etc.), electronic content provided at an audio output device, electronic content provided at a haptic feedback device, connected devices (e.g., temperature control devices, light control devices, speakers, etc.), or other objects. Examples of intentions can include desired adjustments to operational states of devices (e.g., turn off device, turn on device, adjust device brightness, adjust device output volume, etc.), desired interactions with electronically-provided content (e.g., select object, select menu item, navigate to another web page, scroll up, scroll down, close window, etc.), desired interactions with a virtual assistant, or any other intentions.

As such, in one specific example, the camera subsystem925, in combination with other system outputs, can cooperate to determine that the user is looking at a particular connected light in the user's bedroom, decode a brain activity signal that indicates that the user wants to dim the light, and generate control instructions for dimming the light, all without the user speaking a command or adjusting dimness of the light using a physically-manipulated controller. In another specific example, the camera subsystem925, in combination with other system outputs, can cooperate to determine that the user is looking at a selectable button for purchasing an item within an online marketplace, decode a brain activity signal that indicates that the user wants to “click the button”, and generate control instructions for selecting the button to purchase the item, all without the user speaking a command or physically clicking the button (e.g., with a mouse).

In relation to the additional sensors140ashown inFIG.1A, the system can additionally or alternatively include other sensors and/or biometric sensors for sensing aspects of the user, the user's physiology, and/or the environment of the user. Other sensors can include audio sensors (e.g., microphones), motion/orientation sensors (e.g., accelerometers, gyroscopes, inertial measurement units, etc.), respiration sensors (e.g., plethysmography sensors), cardiovascular sensors (e.g., electrical signal-based cardiovascular sensors, radar-based cardiovascular sensors, force-based cardiovascular sensors, etc.), temperature sensors for monitoring environmental temperature (e.g., ambient temperature) and/or body temperature of the user, other brain activity sensors (e.g., electroencephalography sensors), other electrophysiology sensors (e.g., skin conductance sensors), and/or any other suitable sensors.

Outputs of the additional sensors140acan be processed with outputs of other system components described above, in order to improve applications where co-processing brain activity information with other sensor-derived information would be beneficial.

The system can include additional electronics coupled to one or more of the embodiments of the emitter, detector array, additional sensors, network, and/or wearable interface, as described above.

For instance, the system can include a power component that provides power and/or manages power provision to one or more other system components. The power component can include a battery (e.g., rechargeable battery, non-rechargeable battery) electrically coupled to a power management system that maintains desired circuit voltages and/or current draw appropriate for different system components. The power component can be retained within a housing (e.g., enclosures245and255of the BCI module200) associated with the wearable interface and coupled to the emitter array and/or the detector array. As described in relation to other system components below, the housing can house one or more of: the emitter array, the detector array, a power component, a computing component, a data link, and additional sensors. The housing can also house at least a portion of the BCI interface that is head-mountable for positioning signal transmission components at the head region of a user. The housing can be head-mounted or can be coupled to the user in another manner. The housing can be composed of a polymer material and/or any other suitable materials.

As shown inFIG.1A, the system can also include a computing device160that functions to coordinate light transmission from the emitter array and/or operation states of the detector array (e.g., in relation to emission from the emitter array). The computing component can thus include architecture storing instructions in non-transitory computer readable media for implementing portions of methods described, controlling operation states of the emitter array, the detector array, and/or additional sensors, monitoring states of components coupled to the computing component, storing data in memory, coordinating data transfer (e.g., in relation to the data link described below), and/or performing any other suitable computing function of the system. The computing component can additionally or alternatively include signal conditioning elements (e.g., amplifiers, filters, analog-to-digital converters, digital-to-analog converters, etc.) for processing signal outputs of sensors of the system.

The system can also include a data link coupled to the computing device, for handling data transfer between electronics of the wearable system components and the network170a. The data link can provide a wired and/or wireless (e.g., WiFi, Bluetooth LE, etc.) interface with the network or other external systems.

FIG.10Aillustrates a flow chart of a method1000for neural decoding, according to an embodiment.FIG.10Billustrates a flow diagram for an embodiment of the method for neural decoding shown inFIG.10A, according to an embodiment. As shown inFIGS.10A and10B, the system (e.g., light transmission, light detection, and computing components) generates1010a neural data stream capturing brain activity user in response to detecting a set of light signals from a head region of a user as the user interacts with an object in an environment. The system extracts1020a predicted user action upon processing the neural data stream with a neural decoding model, where the predicted user action can be actual (e.g., actual speech) or imagined (e.g., thought), as described in more detail below. The system then provides1030a feedback stimulus to the user based upon the predicted user action, and generates an updated neural decoding model based upon a response of the user to the feedback stimulus1040. The system also implements one or more co-learning processes1040,1050for improvement of the neural decoding model and/or behavior of the user. As such, the method1000can provide a closed loop process whereby the neural decoding model is updated and trained as the user interacts with content or other stimuli, and provides additional light-derived signals that capture brain activity.

The method1000functions to rapidly (e.g., in real time or near real time) decode light-derived signals to extract predicted user actions or intents (e.g., commands) in relation to interactions with objects (e.g., virtual objects, physical objects), such that the user can manipulate the objects or otherwise receive assistance without manually interacting with an input device (e.g., touch input device, audio input device, etc.). The method1000thus provides a neural decoding process with the neural stream signal as an input, and provides feedback to the user, where the feedback is used to train the neural decoding algorithm and user behavior. The neural signals can be blood oxygenation level dependent (BOLD) signals associated with activation of different articulators of the motor cortex, and signals can characterize both actual and imagined motor cortex-related behaviors. With training of the decoding algorithm, rapid calibration of the system for new users can additionally be achieved.

As shown inFIGS.10A and10B, the BCI system (e.g., light transmission, light detection, and computing components) generates1010a neural data stream capturing brain activity of the user as the user interacts with an object, which functions to generate source data that can be processed with the neural decoding model to decode cognition through a non-traditional method. As noted above, the neural data stream is derived from input light signals that are provided to the user's head and output light signals that are captured after passing through the user's head, where the output signals carry information about the level of oxygen present in blood of the user, associated with different regions. As such, the signals associated with the neural data stream are a type of blood oxygen-level dependent (BOLD) signal that carries hemodynamic response information. In use, signals generated can be evaluated for reliability, such that only reliable signals are passed by the controller through downstream processing steps to generate predicted user actions. Reliability can be evaluated based upon consistency in signal characteristics (e.g., variances around a mean signal characteristic).

As described above in relation to the detector array, in generating the neural data stream, the BCI system can transform input signals from a detector-associated region of the head to a brain region-associated space (e.g., in relation to brain regions associated with the input signals). Also described above in relation to embodiments of the detector array, signals of the neural data stream that are derived from light emitted at distinct head regions, where features can be extracted from saturated central regions and/or unsaturated edge regions. The signals derived from saturated portions can include information related to positions of boundaries between a saturated central region and unsaturated edge regions (indicative of total power), diameter of a saturated central region, projected area of a saturated central region, or any other suitable shape-related features associated with the saturated central region. Features related to unsaturated edge regions can include positions of boundaries between unsaturated edge regions of the, slope related features (e.g., rates of decay) of a heel portion of an unsaturated edge region, features related to integrated areas under an intensity curve corresponding to unsaturated edge regions, or any other suitable shape-related features associated with the unsaturated edge region. While signal characteristics associated with intensity are described above, signal characteristics that can be derived from the generated light signals can include features of any other suitable light-related parameter.

Also described above in relation to the detector array, the system can generate signals of the neural data stream associated with multiple light exposure levels (e.g., short and long exposure levels), different light power settings (e.g., a low power setting, a high power setting), and/or different light parameter settings.

In relation to the detector array described above, the neural data stream includes data from separate head regions (e.g., separate head regions associated with a single cortex, separate head regions associated with multiple cortices) of the user. In one embodiment, the neural data stream includes light data from different head regions, where signals from the different head regions map to a set of motor cortex articulators associated with articulation of different speech components. Articulation can be actual or imagined, such that the signals can carry information associated with actual or imagined speech. Furthermore, with repeated iterations of the method1400, the controller can generate a template that refines mapping between the detector array components and specific brain anatomy of the user(s) associated with the method1000. Over time, aggregation and processing of large amounts of data from the user(s) can be used to provide rapid calibration of system components and system response for the user(s) and/or new users.

FIG.10Cdepicts a schematic of a neural data stream capturing information associated with different articulators, in relation to an embodiment of the method shown inFIG.10A. In the schematic shown inFIG.10C, the neural data stream contains information from different brain/head regions that map to different articulators, where the different articulators are associated with different speech components (e.g., phonemes). In more detail, the neural data stream captures data associated with a first set of light signals corresponding to a first articulator, where the first articulator is a labial articulator associated with the phonemes “p”, “b”, and “m”. The neural data stream also captures data associated with a second set of light signals corresponding to a second articulator, where the second articulator is a labiodental articulator associated with the phonemes “f” and “v”. The neural data stream also captures data associated with a third set of light signals corresponding to a third articulator, where the third articulator is a dental articulator associated with the phoneme “th”. The neural data stream also captures data associated with a fourth set of light signals corresponding to a fourth articulator, where the fourth articulator is an alveolar articulator associated with the phonemes “t”, “d”, “n”, “s”, and “z”. The neural data stream also captures data associated with a fifth set of light signals corresponding to a fifth articulator, where the fifth articulator is a postalveolar articulator associated with the phonemes “sh” and “ch”. The neural data stream also captures data associated with a sixth set of light signals corresponding to a sixth articulator, where the sixth articulator is a velar articulator associated with the phonemes “k”, “g”, and “ng”. The neural data stream also captures data associated with a seventh set of light signals corresponding to a seventh articulator, where the seventh articulator is a glottal articulator associated with the phoneme “h”. In alternative embodiments, however, the neural data stream can additionally or alternatively capture data associated with light signals corresponding to different articulators and/or different speech components. In relation to generation of the neural data stream, the detector array can be configured to separate detector subregions associated with different articulators, in order to increase distinction between signals associated with different articulators.

In relation to signals of the neural data stream shown inFIG.10C, the signals associated with different articulators can be aggregated and processed, as described in downstream portions of the method1000, in order to generate higher bandwidth signals from lower bandwidth carriers associated with articulator-specific signals received at different time points. As such, sequences of activation of different articulators associated with actual or imagined speech can generate low bandwidth carrier signals that can be processed, in relation to temporal and spatial factors, to produce a higher bandwidth signal that can be decoded.

In other embodiments, the system can generate a neural data stream using other techniques including any or more of: functional magnetic resonance imaging (fMRI), other forms of blood-oxygen-level dependent (BOLD) contrast imaging, near-infrared spectroscopy (NIRS), magnetoencephalography (MEG), electrocorticography (ECoG), electroencephalography (EEG), positron emission tomography, nuclear magnetic resonance (NMR) spectroscopy, single-photon emission computed tomography.

5.2 Method—Extracting Predicted User Action

FIG.10Ddepicts a flow chart of a portion of the method for neural decoding shown inFIG.10A. As shown inFIG.10D, the system (e.g., computing components) extracts1020a predicted user action upon processing the neural data stream with a neural decoding model, which functions to transform captured input signals associated with the neural data stream into decoded information that can be used to trigger responses (e.g., by environmental objects) that benefit the user in some manner. The predicted user action, as described above, can include a prediction of actual or imagined speech of the user, where the speech is associated with commands provided by the user to manipulate one or more objects in the environment of the user. The objects can be associated with a virtual or a real environment. For instance, the object can be a player or other entity in a virtual game environment, where the predicted user action is an action that manipulates behavior (e.g., movement) of the player or entity within the virtual game environment. In another example, the object can be a digital object associated with a virtual assistant, such that the predicted user action is an action that commands the virtual assistant to perform a task (e.g., in relation to scheduling, in relation to device operation state manipulation, in relation to executing communications with entities associated with the user, etc.). In another example, the object can be a connected object (e.g., a smart home light, smart home thermostat, smart home speaker, smart home appliance, other smart home device, etc.), such that the predicted user action is an action that affects operation of the connected object, through provision of control instructions to the connected object. In alternative embodiments, however, the predicted user action can be an action associated with different motor cortex functions (e.g., actual or imagined movement of another part of the body), different cognitive functions, different cognitive states (affective states, etc.). The action can, however, be another suitable action.

In relation to extracting predicted user actions, the BCI system (e.g., computing components of the system) can implement a neural decoding model that decodes the probability of a predicted action based upon environment/object state and an analysis of information from the neural data stream. As such, the system, as shown inFIG.10D, can perform decoding by determining1055an empirical probability of an action due to state of the object and/or environment associated with the object, and by determining1060a light signal-decoded probability as determined from the neural data stream. In one embodiment, the probability function can be assumed to have the shape:

p(predicted user action|environment state, neural data)=softmax[α*Q(predicted user action environment state)+β*L(predicted user action neural data)], where p is the probability of the predicted user action. Q is determined from an analysis of probability of a given action from a set of candidate options based upon the environment or object state. L is a negative log-likelihood given by an estimator (e.g., neural network model) that is trained based upon incoming neural data from one or more users. The parameters α and β are free hyperparameters.

In an example associated with a gaming environment, where the goal is to navigate a grid to drive a character toward a prize positioned within the grid, Q is determined by policy iteration over the available positions on the grid. In more detail, Q in this example is the negative of the distance from the position of the character to the position of the prize, where the distance can be measured by Dijkstra's algorithm or another distance-determining algorithm. In different environments (e.g., other virtual or real environments) with different objects, however, Q can be used to determine a probability of an action based on environment state with another process.

In an example where the predicted user action is associated with actual or imagined speech commands, the neural decoding model can determine L upon receiving and aggregating sequences of signals associated with the speech articulators, in order to form single or multi-consonant words that represent different commands. As such, the neural decoding model can transform the neural data stream into a set of speech components mapped to a set of motor cortex articulators associated with the head region or, can transform a sequence of activated motor cortex articulators, captured in the set of light signals of the neural data stream, into one or more phoneme chains representative of the commands. Based on the set of articulators from which signals are able to be captured by the detector array, the phoneme chains can be literally translated into the commands (e.g., phoneme chains that form directional words). Additionally or alternatively, the phoneme chains can be trained representations of a spoken version of the commands. For instance, a phoneme chain of “h” “th” “m” detected in the neural data stream can translate to “hotham”, a phoneme chain of “v” “d” “k” detected in the neural data stream can translate to “vadok”, a phoneme chain of “p” “ch” “th” detected in the neural data stream can translate to “poochoth”, and a phoneme chain of “k” “v” “n” detected in the neural data stream can translate to “kevin”, where the representative words “hotham”, “vadok”, “poochoth”, and “kevin” map to different commands (e.g., commands that move a character in different directions, such as left, right, up, and down, in a virtual environment).

Also shown inFIG.10D, implementation of the neural decoding model can include modulating which components (e.g., an environment state-based component, a neural data stream-based component) govern the output of the model. In more detail, as shown inFIG.14C, the controller can determine1055an environment state-based probability associated with the predicted user action and determine1060a light signal-decoded probability associated with the user action. The controller can also determine1065a value of a confidence parameter associated with the light signal-decoded probability, and compare1070the value of the confidence parameter to a threshold condition. Then, if the threshold condition is satisfied, the output of the neural decoding model can be based upon the light signal-decoded data (e.g., as determined using a diffuse optical tomography analysis). However, if the threshold condition is not satisfied, the output of the neural decoding model can be based upon the environment state-based probability analysis. Alternatively, the controller can implement a weighting function that weighs the confidences in each of the environment state-based analyses and the light-signal decoded analyses, and provides an output that combines the weighted probability components as an aggregated output (e.g., based on convex combination, based on another combination algorithm).

As such, without knowledge by the user, the controller can ensure that the neural decoding model1020outputs a predicted user action, even if the confidence in the analysis of the action captured in the neural data stream is low, by using an empirical probability determined from the environment state. Furthermore, the controller can implement training data from situations where the predicted user action is known, in order to increase the accuracy of the light-decoded probabilities. Then, as the confidence in the light signal-decoded probability based on analysis of the neural data stream increases, the controller can primarily output predicted user actions based on analysis of the neural data stream, as the accuracy in decoding light signals of the neural data stream increases.

FIG.10Edepicts an expanded view of a portion of the process flow shown inFIG.10D, in relation to implementation of the neural decoding model as applied to input signals derived from sensors associated with the brain computer interface. As shown, input signals can be associated with hemodynamic responses captured in light signals. Input signals can additionally or alternatively include local field potentials, signals capturing speech processes (actual or imagined speech processes), signals capturing oculomotor processes, signals capturing other motor processes, other biometric data (e.g., associated with respiration, etc.) and other suitable input signals, based on sensor outputs of the BCI. In particular, an embodiment of the detector array can generate light signals associated with hemodynamic response in relation to different articulators, as described above. Additionally, one or more microphones can generate audio signals (e.g., capturing speech information) that can supplement data used to generate the predicted user action. Additionally, one or more video cameras can generate video data associated with the user's face or eyes and/or an environment of the user that can supplement data used to generate the predicted user action. Additionally, one or more touch sensors can generate signals indicative of motor skill activities of the user that can supplement data used to generate the predicted user action. Additionally, one or more motion sensors (e.g., of an inertial measurement unit) can generate signals indicative of motion of the user that can supplement data used to generate the predicted user action. Additionally, other brain activity sensors (e.g., electrodes for electroencephalography, etc.) can generate signals from electrical potentials that can supplement data used to generate the predicted user action. Additionally, other biometric sensors (e.g., respiration sensors, cardiovascular parameter sensors, etc.) can generate signals that can supplement data used to generate the predicted user action.

In relation to processing of the neural data stream, input light signals derived from one or more embodiments of the detector array described above can be classified by the controller hosting the neural decoding model into groups associated with different articulators (e.g., different articulators associated with different speech components, an example of which is shown inFIG.14C). Then, outputs of the classification can be assembled (e.g., into n-grams), based upon temporal factors or other factors. The controller hosting the neural decoding model can then process the assembled information with an action prediction model. In one embodiment, the controller can transform signals associated with a sequence of activated motor cortex articulators (e.g., as captured in a set of light signals) into a phoneme chain representative of a command intended to be executed by the user.

If including analysis of audio signals in the neural decoding model, the controller hosting the neural decoding model can also extract features of the audio signals, determine and annotate speech components or other audio components from the audio signals, and perform a segmentation operation to determine boundaries between individual speech components, in relation to a user action. If including analysis of video signals in the neural decoding model, the controller hosting the neural decoding model can also implement facial feature tracking algorithms, with fiducial labeling and gesture segmentation models, in relation to detecting a user action. The controller can additionally or alternatively process video signals in order to track motions of the eye(s) of the user, in order to determine coordinates of objects that the user is looking at and/or dwell time, in relation to a user action. The neural decoding model can additionally or alternatively accept other input signals, which can be aggregated by architecture of the neural decoding model to combine features into an output of a predicted user action (e.g., with a confidence score).

In relation to the neural decoding model, the controller can include architecture for synchronization of input signals associated with the same or different sensors, in relation to an event (e.g., an environment state, an object state, a stimulus, a feedback stimulus provided based on a predicted user action, as described in more detail below, etc.). In order to synchronize input signals, the controller can include architecture for signal registration (e.g., based upon temporal signatures within different signals, based upon interpolation of signals with different associated sampling rates, etc.), to a desired degree (e.g., with millisecond alignment, with microsecond alignment, etc.). As such, the controller implements the neural decoding model to extract predicted user actions contemporaneously (e.g., within a time threshold to) a time point associated with an event, such as a state of an environment or object associated with the user.

As shown inFIGS.10A,10D, and10F, whereFIG.10Fillustrates an expanded view of a portion of the process flow shown inFIG.10B, the controller provides1030a feedback stimulus to the user based on the predicted user action output by the neural decoding model. The feedback stimulus can be a representation of the predicted user action output by the neural decoding model, in text or other visual format, in audio format, and/or in haptic format. For instance, if the predicted user action is associated with a command (e.g., a command to manipulate an object) or request, the feedback stimulus can be a textual representation of the command or request, or a symbolic representation of the command or request. In a specific example, if the command is an indication by the user that the user wants to move an object in a direction, the feedback stimulus can be a rendered text description of the direction or a rendered arrow depicting the direction, where the controller generates instructions for rendering the feedback stimulus at a display. In another specific example, if the command is an indication by the user that the user wants to move an object in a direction, the feedback stimulus can be an audio output that states the direction in speech, where the controller generates instructions for transmitting audio through a speaker of a device associated with the user. The representation of the command, provided to the user as the feedback stimulus, can be validated by the user (e.g., the user can indicate that the predicted user action is correct, based upon the feedback stimulus), as a transitional step to execution of the command or request by the controller or other device. Additionally, as described below, the representation of the command, provided to the user as the feedback stimulus, can be used in a co-learning process in order to train the user's behavior (e.g., to provide feedback to the user so that the user can tune his/her behaviors to provide signals that are more easily decoded), such that training of the neural decoding model occurs in coordination with training of user behaviors to increase the accuracy of the neural decoding model.

In providing the feedback stimulus, the controller can also generate instructions for execution of a command or request by the user, in relation to modulation of a state of a digital object1085or a physical object1090, with generation1080of control instructions in a computer-readable medium for object modulation, several examples of which are described below.

In the context of a game architected in a digital platform, the feedback stimulus can include direct manipulation of a user's character in the game, in terms of motion, behavior, or another action performable by the user's character. In a specific example, execution of the command can include moving the user's character in the game environment, in direct response to the predicted user action being associated with a direction in which the user intends the character to move. In the context of a game architected in a digital platform, the feedback stimulus can include direct manipulation of a game environment, in terms of adjustable parameters in the virtual environment.

In the context of a virtual assistant platform, the feedback stimulus can include generation of control instructions for the virtual assistant to navigate and/or manipulate systems in order to perform a task for the user. For instance, the controller can generate control instructions that instruct the virtual assistant to execute communication (e.g., in a text message, in an audio message, etc.) with an entity associated with the user, to generate a reminder, to perform a calendar-related task, or to perform another task.

In the context of a virtual environment, with menus or other selectable objects, the feedback stimulus can include execution of instructions for selection of the object or navigation of a menu, based upon the predicted user action.

In the context of connected devices physically associated with a real environment of the user, the feedback stimulus can include manipulation of operation states of the connected device(s). In examples, the connected device(s) can include one or more of: temperature control devices, light control devices, speakers, locks, appliances, and other connected devices. In providing the feedback stimulus, the controller can generate control instructions for adjusting operational states of devices (e.g., turn off device, turn on device, transition device to idle, adjust device brightness, adjust device output color, adjust device output volume, adjust device sound output profile, adjust microphone operation state, adjust temperature output, adjust lock state, adjust appliance operation state, etc.)

In other contexts, the feedback stimulus may not be related to a command or request. For instance, the predicted user action can be a subconscious cognitive state or affective state, and the controller can generate and/or execute instructions for manipulation of an object or environment based upon the subconscious cognitive state or affective state.

As shown inFIGS.10A and10Band described above, the system also implements one or more co-learning processes1040,1050for improvement of the neural decoding model and/or behavior of the user. In relation to the co-learning processes, the controller implementing the method1400provides a closed loop process whereby the neural decoding model is updated and trained as the user interacts with content or other stimuli, and provides additional light-derived signals that capture brain activity. Additionally, the feedback stimuli provided to the user produces a behavior by the user and can be used to train the user in relation to adjusting responses to the environment in a manner that is more efficiently decoded by the neural decoding model. As such, the method1400can implement computing architecture that inputs an output derived from the feedback stimulus (e.g., verification that the feedback stimulus was appropriate in relation to a user action) back into the neural decoding model, in order to refine the neural decoding model. In relation to a closed-loop system, the controller can, based upon a behavior of the user in response to the feedback stimulus, process additional signals from the user to generate additional predicted user actions (with refinement of the neural decoding model). Then, based upon the additional predicted user actions, the controller can provide additional feedback stimuli to the user, derivatives of which and/or responses to which can be used to further refine the neural decoding model.