Patent ID: 12251233

DETAILED DESCRIPTION

Functional near infrared spectroscopy (fNIRS) is a brain imaging modality that allows the indirect inference of cortical responses (a proxy for neural activity) by measuring the hemodynamic response of the brain tissue in different cortical regions. Compared to function magnetic resonance imaging (fMRI), another imaging modality based on the hemodynamic response, fNIRS can be mobile and low cost, allowing the large-scale use of imaging technology to study the brain, either within a tightly constrained environment of a scientific lab or in real-world scenarios, where subjects interact freely with their environment. One disadvantage of fNIRS, however, is that it is slow compared to the temporal dynamics of the current dipoles produced by neurons firing in the cortex. Another drawback of fNIRS is that there is no unique inverse mapping (ill-posed problem) from the measurements onto the consumption of oxyhemoglobin (OHb) and deoxyhemoglobin (HHb) molecules in different parts of the cortex. One way to solve this inverse problem is to introduce mathematical constraints into the reconstruction algorithm. Furthermore, fNIRS source reconstruction is typically performed offline using standard inverse solvers not optimized for this type of data.

Another low-cost and mobile brain imaging modality is electroencephalogram (EEG). Compared with fNIRS, EEG devices have a better temporal resolution because they use scalp sensors (e.g., electrodes) to measure the macroscopic electrical activity generated by clusters of cortical neurons that synchronize in space and time. Like fNIRS, the inverse mapping from EEG voltage sensors onto cortical current dipoles is not unique, therefore, to solve this inverse problem constraints are also needed. As such, fNIRS and EEG data fusion is attractive for brain imaging because each modality can complement each other by bringing data-driven constraints into an inverse mapping algorithm in which each constraint is enforced at the right spatiotemporal scales.

Accordingly, multimodal wearable measurement systems that include both optical and electrical activity measurement components are described herein. An exemplary multimodal measurement system includes a wearable assembly configured to be worn by a user and comprising a plurality of light sources each configured to emit light directed at a target within the user, a plurality of detectors configured to detect arrival times for photons of the light after the light is scattered by the target, and a plurality of electrodes configured to be external to the user and detect electrical activity of the target. In some examples, the multimodal measurement system further includes a processing unit configured to generate optical measurement data based on the arrival times detected by the detectors and electrical measurement data based on the electrical activity detected by the electrodes. The processing unit may be further configured to process the optical measurement data and the electrical measurement data (e.g., in real time during operation of the detectors and electrodes) in accordance with a data fusion heuristic to generate an estimate of cortical source activity and/or otherwise determine one or more other physiological characteristics of a user.

The systems and methods described herein may provide various benefits. For example, the systems and methods described herein may be optimized for brain tomography based on a computationally efficient fusion of optical measurement data (e.g., fNIRS data) and electrical measurement data (e.g., EEG data). This fusion method may benefit brain mapping algorithms by providing data-driven spatiotemporal constraints at different temporal scales. In particular, the systems and methods described herein may allow for computationally efficient real-time imaging. This technology has the potential to improve basic neuro-scientific research as well as the development of imaging-based translational neurotechnologies, such as brain-computer interfaces (BCIs) and continuous brain monitoring.

These and other advantages and benefits of the present systems and methods are described more fully herein and/or will be made apparent in the description herein.

FIG.1shows an exemplary optical measurement system100configured to perform an optical measurement operation with respect to a body102. Optical measurement system100may, in some examples, be portable and/or wearable by a user.

In some examples, optical measurement operations performed by optical measurement system100are associated with a time domain-based optical measurement technique. Example time domain-based optical measurement techniques include, but are not limited to, TCSPC, time domain near infrared spectroscopy (TD-NIRS), time domain diffusive correlation spectroscopy (TD-DCS), and time domain digital optical tomography (TD-DOT).

As shown, optical measurement system100includes a detector104that includes a plurality of individual photodetectors (e.g., photodetector106), a processor108coupled to detector104, a light source110, a controller112, and optical conduits114and116(e.g., light guides, as described more fully herein). However, one or more of these components may not, in certain embodiments, be considered to be a part of optical measurement system100. For example, in implementations where optical measurement system100is wearable by a user, processor108and/or controller112may in some embodiments be separate from optical measurement system100and not configured to be worn by the user.

Detector104may include any number of photodetectors106as may serve a particular implementation, such as 2nphotodetectors (e.g., 256, 512, . . . , 16384, etc.), where n is an integer greater than or equal to one (e.g., 4, 5, 8, 10, 11, 14, etc.). Photodetectors106may be arranged in any suitable manner.

Photodetectors106may each be implemented by any suitable circuit configured to detect individual photons of light incident upon photodetectors106. For example, each photodetector106may be implemented by a single photon avalanche diode (SPAD) circuit and/or other circuitry as may serve a particular implementation.

Processor108may be implemented by one or more physical processing (e.g., computing) devices. In some examples, processor108may execute instructions (e.g., software) configured to perform one or more of the operations described herein.

Light source110may be implemented by any suitable component configured to generate and emit light. For example, light source110may be implemented by one or more laser diodes, distributed feedback (DFB) lasers, super luminescent diodes (SLDs), light emitting diodes (LEDs), diode-pumped solid-state (DPSS) lasers, super luminescent light emitting diode (sLEDs), vertical-cavity surface-emitting lasers (VCSELs), titanium sapphire lasers, a micro light emitting diodes (mLEDs), and/or any other suitable laser or light source configured to emit light in one or more discrete wavelengths or narrow wavelength bands. In some examples, the light emitted by light source110is high coherence light (e.g., light that has a coherence length of at least 5 centimeters) at a predetermined center wavelength. In some examples, the light emitted by light source110is emitted as a plurality of alternating light pulses of different wavelengths.

Light source110is controlled by controller112, which may be implemented by any suitable computing device (e.g., processor108), integrated circuit, and/or combination of hardware and/or software as may serve a particular implementation. In some examples, controller112is configured to control light source110by turning light source110on and off and/or setting an intensity of light generated by light source110. Controller112may be manually operated by a user, or may be programmed to control light source110automatically.

Light emitted by light source110travels via an optical conduit114(e.g., a light pipe, a light guide, a waveguide, a single-mode optical fiber, and/or or a multi-mode optical fiber) to body102of a subject. Body102may include any suitable turbid medium. For example, in some implementations, body102is a head or any other body part of a human or other animal. Alternatively, body102may be a non-living object. For illustrative purposes, it will be assumed in the examples provided herein that body102is a human head.

As indicated by arrow120, light emitted by light source110enters body102at a first location122on body102. Accordingly, a distal end of optical conduit114may be positioned at (e.g., right above, in physical contact with, or physically attached to) first location122(e.g., to a scalp of the subject). In some examples, the light may emerge from optical conduit114and spread out to a certain spot size on body102to fall under a predetermined safety limit. At least a portion of the light indicated by arrow120may be scattered within body102.

As used herein, “distal” means nearer, along the optical path of the light emitted by light source110or the light received by detector104, to the target (e.g., within body102) than to light source110or detector104. Thus, the distal end of optical conduit114is nearer to body102than to light source110, and the distal end of optical conduit116is nearer to body102than to detector104. Additionally, as used herein, “proximal” means nearer, along the optical path of the light emitted by light source110or the light received by detector104, to light source110or detector104than to body102. Thus, the proximal end of optical conduit114is nearer to light source110than to body102, and the proximal end of optical conduit116is nearer to detector104than to body102.

As shown, the distal end of optical conduit116(e.g., a light pipe, a light guide, a waveguide, a single-mode optical fiber, and/or a multi-mode optical fiber) is positioned at (e.g., right above, in physical contact with, or physically attached to) output location126on body102. In this manner, optical conduit116may collect at least a portion of the scattered light (indicated as light124) as it exits body102at location126and carry light124to detector104. Light124may pass through one or more lenses and/or other optical elements (not shown) that direct light124onto each of the photodetectors106included in detector104.

Photodetectors106may be connected in parallel in detector104. An output of each of photodetectors106may be accumulated to generate an accumulated output of detector104. Processor108may receive the accumulated output and determine, based on the accumulated output, a temporal distribution of photons detected by photodetectors106. Processor108may then generate, based on the temporal distribution, a histogram representing a light pulse response of a target (e.g., tissue, blood flow, etc.) in body102. Example embodiments of accumulated outputs are described herein.

FIG.2illustrates an exemplary detector architecture200that may be used in accordance with the systems and methods described herein. As shown, architecture200includes a SPAD circuit202that implements photodetector106, a control circuit204, a time-to-digital converter (TDC)206, and a signal processing circuit208. Architecture200may include additional or alternative components as may serve a particular implementation.

In some examples, SPAD circuit202includes a SPAD and a fast gating circuit configured to operate together to detect a photon incident upon the SPAD. As described herein, SPAD circuit202may generate an output when SPAD circuit202detects a photon.

The fast gating circuit included in SPAD circuit202may be implemented in any suitable manner. For example, the fast gating circuit may include a capacitor that is pre-charged with a bias voltage before a command is provided to arm the SPAD. Gating the SPAD with a capacitor instead of with an active voltage source, such as is done in some conventional SPAD architectures, has a number of advantages and benefits. For example, a SPAD that is gated with a capacitor may be armed practically instantaneously compared to a SPAD that is gated with an active voltage source. This is because the capacitor is already charged with the bias voltage when a command is provided to arm the SPAD. This is described more fully in U.S. Pat. Nos. 10,158,038 and 10,424,683, which are incorporated herein by reference in their entireties.

In some alternative configurations, SPAD circuit202does not include a fast gating circuit. In these configurations, the SPAD included in SPAD circuit202may be gated in any suitable manner.

Control circuit204may be implemented by an application specific integrated circuit (ASIC) or any other suitable circuit configured to control an operation of various components within SPAD circuit202. For example, control circuit204may output control logic that puts the SPAD included in SPAD circuit202in either an armed or a disarmed state.

In some examples, control circuit204may control a gate delay, which specifies a predetermined amount of time control circuit204is to wait after an occurrence of a light pulse (e.g., a laser pulse) to put the SPAD in the armed state. To this end, control circuit204may receive light pulse timing information, which indicates a time at which a light pulse occurs (e.g., a time at which the light pulse is applied to body102). Control circuit204may also control a programmable gate width, which specifies how long the SPAD is kept in the armed state before being disarmed.

Control circuit204is further configured to control signal processing circuit208. For example, control circuit204may provide histogram parameters (e.g., time bins, number of light pulses, type of histogram, etc.) to signal processing circuit208. Signal processing circuit208may generate histogram data in accordance with the histogram parameters. In some examples, control circuit204is at least partially implemented by controller112.

TDC206is configured to measure a time difference between an occurrence of an output pulse generated by SPAD circuit202and an occurrence of a light pulse. To this end, TDC206may also receive the same light pulse timing information that control circuit204receives. TDC206may be implemented by any suitable circuitry as may serve a particular implementation.

Signal processing circuit208is configured to perform one or more signal processing operations on data output by TDC206. For example, signal processing circuit208may generate histogram data based on the data output by TDC206and in accordance with histogram parameters provided by control circuit204. To illustrate, signal processing circuit208may generate, store, transmit, compress, analyze, decode, and/or otherwise process histograms based on the data output by TDC206. In some examples, signal processing circuit208may provide processed data to control circuit204, which may use the processed data in any suitable manner. In some examples, signal processing circuit208is at least partially implemented by processor108.

In some examples, each photodetector106(e.g., SPAD circuit202) may have a dedicated TDC206associated therewith. For example, for an array of N photodetectors106, there may be a corresponding array of N TDCs206. Alternatively, a single TDC206may be associated with multiple photodetectors106. Likewise, a single control circuit204and a single signal processing circuit208may be provided for a one or more photodetectors106and/or TDCs206.

FIG.3illustrates an exemplary timing diagram300for performing an optical measurement operation using optical measurement system100. The optical measurement operation may be performed in accordance with a time domain-based technique, such as TD-NIRS. Optical measurement system100may be configured to perform the optical measurement operation by directing light pulses (e.g., laser pulses) toward a target within a body (e.g., body102). The light pulses may be short (e.g., 10-2000 picoseconds (ps)) and repeated at a high frequency (e.g., between 100,000 hertz (Hz) and 100 megahertz (MHz)). The light pulses may be scattered by the target and at least a portion of the scattered light may be detected by optical measurement system100. Optical measurement system100may measure a time relative to the light pulse for each detected photon. By counting the number of photons detected at each time relative to each light pulse repeated over a plurality of light pulses, optical measurement system100may generate a histogram that represents a light pulse response of the target (e.g., a temporal point spread function (TPSF)). The terms histogram and TPSF are used interchangeably herein to refer to a light pulse response of a target.

Timing diagram300shows a sequence of light pulses302(e.g., light pulses302-1and302-2) that may be applied to the target (e.g., tissue within a brain of a user, blood flow, a fluorescent material used as a probe in a body of a user, etc.). Timing diagram300also shows a pulse wave304representing predetermined gated time windows (also referred as gated time periods) during which photodetectors106are gated ON to detect photons. As shown, light pulse302-1is applied at a time t0. At a time t1, a first instance of the predetermined gated time window begins. Photodetectors106may be armed at time t1, enabling photodetectors106to detect photons scattered by the target during the predetermined gated time window. In this example, time t1is set to be at a certain time after time t0, which may minimize photons detected directly from the laser pulse, before the laser pulse reaches the target. However, in some alternative examples, time t1is set to be equal to time t0.

At a time t2, the predetermined gated time window ends. In some examples, photodetectors106may be disarmed at time t2. In other examples, photodetectors106may be reset (e.g., disarmed and re-armed) at time t2or at a time subsequent to time t2. During the predetermined gated time window, photodetectors106may detect photons scattered by the target. Photodetectors106may be configured to remain armed during the predetermined gated time window such that photodetectors106maintain an output upon detecting a photon during the predetermined gated time window. For example, a photodetector106may detect a photon at a time t3, which is during the predetermined gated time window between times t1and t2. The photodetector106may be configured to provide an output indicating that the photodetector106has detected a photon. The photodetector106may be configured to continue providing the output until time t2, when the photodetector may be disarmed and/or reset. Optical measurement system100may generate an accumulated output from the plurality of photodetectors. Optical measurement system100may sample the accumulated output to determine times at which photons are detected by photodetectors106to generate a TPSF.

FIG.4illustrates a graph400of an exemplary TPSF402that may be generated by optical measurement system100in response to a light pulse404(which, in practice, represents a plurality of light pulses). Graph400shows a normalized count of photons on a y-axis and time bins on an x-axis. As shown, TPSF402is delayed with respect to a temporal occurrence of light pulse404. In some examples, the number of photons detected in each time bin subsequent to each occurrence of light pulse404may be aggregated (e.g., integrated) to generate TPSF402. TPSF402may be analyzed and/or processed in any suitable manner to determine or infer biological (e.g., neural) activity.

Optical measurement system100may be implemented by or included in any suitable device(s). For example, optical measurement system100may be included in a non-wearable device (e.g., a medical device and/or consumer device that is placed near the head or other body part of a user to perform one or more diagnostic, imaging, and/or consumer-related operations). Optical measurement system100may alternatively be included, in whole or in part, in a sub-assembly enclosure of a wearable invasive device (e.g., an implantable medical device for brain recording and imaging).

Alternatively, optical measurement system100may be included, in whole or in part, in a non-invasive wearable device that a user may wear to perform one or more diagnostic, imaging, analytical, and/or consumer-related operations. The non-invasive wearable device may be placed on a user's head or other part of the user to detect neural activity. In some examples, such neural activity may be used to make behavioral and mental state analysis, awareness and predictions for the user.

Mental state described herein refers to the measured neural activity related to physiological brain states and/or mental brain states, e.g., joy, excitement, relaxation, surprise, fear, stress, anxiety, sadness, anger, disgust, contempt, contentment, calmness, focus, attention, approval, creativity, positive or negative reflections/attitude on experiences or the use of objects, etc. Further details on the methods and systems related to a predicted brain state, behavior, preferences, or attitude of the user, and the creation, training, and use of neuromes can be found in U.S. Provisional Patent Application No. 63/047,991, filed Jul. 3, 2020. Exemplary measurement systems and methods using biofeedback for awareness and modulation of mental state are described in more detail in U.S. patent application Ser. No. 16/364,338, filed Mar. 26, 2019, published as US2020/0196932A1. Exemplary measurement systems and methods used for detecting and modulating the mental state of a user using entertainment selections, e.g., music, film/video, are described in more detail in U.S. patent application Ser. No. 16/835,972, filed Mar. 31, 2020, published as US2020/0315510A1. Exemplary measurement systems and methods used for detecting and modulating the mental state of a user using product formulation from, e.g., beverages, food, selective food/drink ingredients, fragrances, and assessment based on product-elicited brain state measurements are described in more detail in U.S. patent application Ser. No. 16/853,614, filed Apr. 20, 2020, published as US2020/0337624A1. Exemplary measurement systems and methods used for detecting and modulating the mental state of a user through awareness of priming effects are described in more detail in U.S. patent application Ser. No. 16/885,596, filed May 28, 2020, published as US2020/0390358A1. These applications and corresponding U.S. publications are incorporated herein by reference in their entirety.

To illustrate,FIG.5shows an exemplary non-invasive wearable brain interface system500(“brain interface system500”) that implements optical measurement system100(shown inFIG.1). As shown, brain interface system500includes a head-mountable component502configured to be attached to and/or worn on a user's head. Head-mountable component502may be implemented by a cap shape that is worn on a head of a user. Alternative implementations of head-mountable component502include helmets, beanies, headbands, other hat shapes, or other forms conformable to be worn on a user's head, etc. Head-mountable component502may be made out of any suitable cloth, soft polymer, plastic, hard shell, and/or any other suitable material as may serve a particular implementation. Examples of headgears used with wearable brain interface systems are described below in more detail and in U.S. Pat. No. 10,340,408, incorporated herein by reference in its entirety.

Head-mountable component502includes a plurality of detectors504, which may implement or be similar to detector104, and a plurality of light sources506, which may be implemented by or be similar to light source110. It will be recognized that in some alternative embodiments, head-mountable component502may include a single detector504and/or a single light source506.

Brain interface system500may be used for controlling an optical path to the brain and/or for transforming photodetector measurements into an intensity value that represents an optical property of a target within the brain. Brain interface system500allows optical detection of deep anatomical locations beyond skin and bone (e.g., skull) by extracting data from photons originating from light sources506and emitted to a target location within the user's brain, in contrast to conventional imaging systems and methods (e.g., optical coherence tomography (OCT), continuous wave near infrared spectroscopy (CW-NIRS)), which only image superficial tissue structures or through optically transparent structures.

Brain interface system500may further include a processor508configured to communicate with (e.g., control and/or receive signals from) detectors504and light sources506by way of a communication link510. Communication link510may include any suitable wired and/or wireless communication link. Processor508may include any suitable housing and may be located on the user's scalp, neck, shoulders, chest, or arm, as may be desirable. In some variations, processor508may be integrated in the same assembly housing as detectors504and light sources506. In some examples, processor508is implemented by or similar to processor108and/or controller112.

As shown, brain interface system500may optionally include a remote processor512in communication with processor508. For example, remote processor512may store measured data from detectors504and/or processor508from previous detection sessions and/or from multiple brain interface systems (not shown). In some examples, remote processor512is implemented by or similar to processor108and/or controller112.

Power for detectors504, light sources506, and/or processor508may be provided via a wearable battery (not shown). In some examples, processor508and the battery may be enclosed in a single housing, and wires carrying power signals from processor508and the battery may extend to detectors504and light sources506. Alternatively, power may be provided wirelessly (e.g., by induction).

In some alternative embodiments, head mountable component502does not include individual light sources. Instead, a light source configured to generate the light that is detected by detector504may be included elsewhere in brain interface system500. For example, a light source may be included in processor508and/or in another wearable or non-wearable device and coupled to head mountable component502through an optical connection.

In some alternative embodiments, head mountable component502does not include individual detectors504. Instead, one or more detectors configured to detect the scattered light from the target may be included elsewhere in brain interface system500. For example, a detector may be included in processor508and/or in another wearable or non-wearable device and coupled to head mountable component502through an optical connection.

FIG.6shows an exemplary multimodal measurement system600in accordance with the principles described herein. Multimodal measurement system600may at least partially implement optical measurement system100and, as shown, includes a wearable assembly602, which includes N light sources604(e.g., light sources604-1through604-N), M detectors606(e.g., detectors606-1through606-M), and X electrodes (e.g., electrodes608-1through608-X). Multimodal measurement system600may include any of the other components of optical measurement system100as may serve a particular implementation. N, M, and X may each be any suitable value (i.e., there may be any number of light sources604, any number of detectors606, and any number of electrodes608included in multimodal measurement system600as may serve a particular implementation).

Light sources604are each configured to emit light (e.g., a sequence of light pulses) and may be implemented by any of the light sources described herein.

Detectors606may each be configured to detect arrival times for photons of the light emitted by one or more light sources604after the light is scattered by the target. For example, a detector606may include a photodetector configured to generate a photodetector output pulse in response to detecting a photon of the light and a TDC configured to record a timestamp symbol in response to an occurrence of the photodetector output pulse, the timestamp symbol representative of an arrival time for the photon (i.e., when the photon is detected by the photodetector). Detectors606may be implemented by any of the detectors described herein.

Electrodes608may be configured to detect electrical activity within a target (e.g., the brain). Such electrical activity may include EEG activity and/or any other suitable type of electrical activity as may serve a particular implementation. In some examples, electrodes608are all conductively coupled to one another to create a single channel that may be used to detect electrical activity. Alternatively, at least one electrode included in electrodes608is conductively isolated from a remaining number of electrodes included in electrodes608to create at least two channels that may be used to detect electrical activity.

Wearable assembly602may be implemented by any of the wearable devices, modular assemblies, and/or wearable units described herein. For example, wearable assembly602may be implemented by a wearable device (e.g., headgear) configured to be worn on a user's head. Wearable assembly602may additionally or alternatively be configured to be worn on any other part of a user's body.

Multimodal measurement system600may be modular in that one or more components of multimodal measurement system600may be removed, changed out, or otherwise modified as may serve a particular implementation. Additionally or alternatively, multimodal measurement system600may be modular such that one or more components of multimodal measurement system600may be housed in a separate housing (e.g., module) and/or may be movable relative to other components. Exemplary modular multimodal measurement systems are described in more detail in U.S. Provisional Patent Application No. 63/081,754, filed Sep. 22, 2020, U.S. Provisional Patent Application No. 63/038,459, filed Jun. 12, 2020, U.S. Provisional Patent Application No. 63/038,468, filed Jun. 12, 2020, U.S. Provisional Patent Application No. 63/038,481, filed Jun. 12, 2020, and U.S. Provisional Patent Application No. 63/064,688, filed Aug. 12, 2020, which applications are incorporated herein by reference in their respective entireties.

To illustrate, various modular assemblies that may implement multimodal measurement system600are described in connection withFIGS.7-9. The modular assemblies described herein are merely illustrative of the many different implementations of multimodal measurement system600that may be realized in accordance with the principles described herein. Each of the modular assemblies described herein may include one or more modules and may be worn on the head or any other suitable body part of the user.

InFIGS.7-9, the illustrated modules may, in some examples, be physically distinct from each other. For example, as described herein, each module may be configured to be removably attached to a wearable assembly (e.g., by being inserted into a different slot of the wearable assembly). This may allow the modular assemblies to conform to three-dimensional surface geometries, such as a user's head.

InFIGS.7-9, each illustrated module may include one or more light sources labeled “S” and a set of detectors each labeled “D”. Some specific light sources and detectors are also referred to by specific reference numbers.

Each light source depicted inFIGS.7-9may be implemented by one or more light sources similar to light source110and may be configured to emit light directed at a target (e.g., the brain).

In some examples, each light source may be implemented by dual (e.g., two) light sources that are co-located (e.g., right next to each other within the same module). For example, a module may include a first light source and a second light source. In this configuration, the first light source may emit light having a first wavelength and the second light source may emit light having a second wavelength different than the first wavelength. This dual light source configuration may be used when it is desired for the multimodal measurement system to concurrently measure or detect different properties. For example, pairs of lights sources operating at different wavelengths may be used to measure the concentrations of oxygenated and deoxygenated hemoglobin, which are at different wavelengths.

Each detector depicted inFIGS.7-9may implement or be similar to detector104and may include a plurality of photodetectors (e.g., SPADs) as well as other circuitry (e.g., TDCs), and may be configured to detect arrival times for photons of the light emitted by one or more light sources after the light is scattered by the target.

FIG.7shows an illustrative modular assembly700that may implement multimodal measurement system600. As shown, modular assembly700includes a plurality of modules702(e.g., modules702-1through702-3). While three modules702are shown to be included in modular assembly700, in alternative configurations, any number of modules702(e.g., a single module up to sixteen or more modules) may be included in modular assembly700. Moreover, while each module702has a hexagonal shape, modules702may alternatively have any other suitable geometry (e.g., in the shape of a pentagon, octagon, square, rectangular, circular, triangular, free-form, etc.).

Each module702includes a light source (e.g., light source704-1of module702-1and light source704-2of module702-2) and a plurality of detectors (e.g., detectors706-1through706-6of module702-1). In the particular implementation shown inFIG.7, each module702includes a single light source and six detectors. Alternatively, each module702may have any other number of light sources (e.g., two light sources) and any other number of detectors.

Each light source (e.g., light source704-1or light source704-2) depicted inFIG.7may be located at a center region of a surface of the light source's corresponding module. For example, light source704-1is located at a center region of a surface708of module702-1. In alternative implementations, a light source of a module may be located away from a center region of the module.

The detectors of a module may be distributed around the light source of the module. For example, detectors706of module702-1are distributed around light source704-1on surface708of module702-1. In some examples, the detectors of a module may all be equidistant from the light source of the same module. In other words, the spacing between a light source (i.e., a distal end portion of a light source optical conduit) and the detectors (i.e., distal end portions of optical conduits for each detector) are maintained at the same fixed distance on each module to ensure homogeneous coverage over specific areas and to facilitate processing of the detected signals. The fixed spacing also provides consistent spatial (lateral and depth) resolution across the target area of interest, e.g., brain tissue. Moreover, maintaining a known distance between the light source, e.g., light emitter, and the detector allows subsequent processing of the detected signals to infer spatial (e.g., depth localization, inverse modeling, etc.) information about the detected signals. Detectors of a module may be alternatively disposed on the module as may serve a particular implementation.

As shown, modular assembly700further includes a plurality of electrodes710(e.g., electrodes710-1through710-3), which may implement electrodes608. Electrodes710may be located at any suitable location that allows electrodes710to be in physical contact with a surface (e.g., the scalp and/or skin) of a body of a user. For example, in modular assembly700, each electrode710is on a module surface configured to face a surface of a user's body when modular assembly700is worn by the user. To illustrate, electrode710-1is on surface708of module702-1. Moreover, in modular assembly700, electrodes710are located in a center region of each module702and surround each module's light source704. Alternative locations and configurations for electrodes710are described herein.

InFIG.7, modules702are shown to be adjacent to and touching one another. Modules702may alternatively be spaced apart from one another. For example,FIGS.8A-8Bshow another modular assembly800that may implement multimodal measurement system600. In modular assembly800, modules702may be configured to be inserted into individual slots802(e.g., slots802-1through802-3, also referred to as cutouts) of a wearable assembly804. In particular,FIG.8Ashows the individual slots802of the wearable assembly804before modules702have been inserted into respective slots802, andFIG.8Bshows wearable assembly804with individual modules702inserted into respective individual slots802.

Wearable assembly804may implement wearable assembly602and may be configured as headgear and/or any other type of device configured to be worn by a user.

As shown inFIG.8A, each slot802is surrounded by a wall (e.g., wall806) such that when modules702are inserted into their respective individual slots802, the walls physically separate modules702one from another. In alternative embodiments, a module (e.g., module702-1) may be in at least partial physical contact with a neighboring module (e.g., module702-2).

As shown inFIGS.8A-8B, wearable assembly804may include a plurality of connecting structures808(e.g., connecting structures808-1through808-3) configured to interconnect each slot802of wearable assembly804. Connecting structures808may be implemented by any suitable connecting mechanisms (e.g., ball joints, hinges, elastic bands, etc.) and/or support members (e.g., support frames, bands, rails, etc.). In some examples, connecting structures808are flexible and/or movable such that modular assembly800may be adjusted to fit a particular body part (e.g., the head). Moreover, with such a configuration, modular assembly800can be adjusted to conform to a 3D (non-planar) surface, such as a user's head, and/or to target a specific region of interest (e.g., a specific region of the brain).

As shown inFIGS.8A-8B, electrodes810(e.g., electrodes810-1through810-3) that implement electrodes608may be located off-module (i.e., not on any of modules702) on connecting structures808. Additionally or alternatively, one or more electrodes may be located off-module on any other structure or component of wearable assembly804as may serve a particular implementation.

FIG.9shows another modular assembly900that may implement multimodal measurement system600. Modular assembly900is similar to modular assembly800, except that in modular assembly900, electrodes (e.g., electrode902) that implement electrodes608are on (e.g., integrated into) each of the light sources and detectors of modules702. The electrodes may be integrated into one or more of sources and detectors of modules702in any suitable manner. For example, the light sources and detectors may be implemented by light guides that have distal ends configured to be in contact with a surface of a body of the user. In this example, the electrodes may be integrated into the light guides themselves.

To illustrate,FIG.10shows a perspective view of a module1000that may implement any of the modules described herein. Module1000is described in more detail in U.S. Provisional Patent Application No. 63/064,688, filed Aug. 12, 2020, the contents of which are incorporated herein by reference in their entirety.

As shown inFIG.10, module1000includes a housing1002and a plurality of light guides1004(e.g., light guides1004-1through1004-7) protruding from an upper surface1006of housing1002. As used herein with reference to module1000, “upper” refers to a side of module1000that faces a target within a body of a user when module1000is worn by the user.

InFIG.10, light guide1004-1is part of a light source assembly included in module1000. As such, light may pass through light guide1004-1towards the target while module1000is being worn by the user. Light guides1004-2through1004-7are parts of detector assemblies included in module1000. As such, light may be received by light guides1004-2through1004-7after the light is scattered by the target.

In some examples, a least a portion of light guides1004are made out of a conductive material, which allows light guides1004themselves to function as the electrodes that implement electrodes608.

To illustrate,FIG.11shows an exemplary light guide assembly1100that may implement any of the light guides1004shown inFIG.10. As shown, light guide assembly1100includes an upper light guide portion1102, a lower light guide portion1104, a spring member1106, and a flange1108in between upper and lower light guide portions1102and1104.FIG.11also depicts a printed circuit board (PCB)1110attached to a proximal end of lower light guide portion1104.

In some examples, lower light guide portion1104, spring member1106, flange1108, and PCB1110are configured to be housed within housing1002of module1000, while upper light guide portion1102is configured to protrude from upper surface1006of housing1002. In this configuration, upper light guide portion1102may be in contact with a surface of a user.

In the example ofFIG.11, upper light guide portion1102and flange1008are made out of a conductive material, which allows a distal end of the upper light guide portion1102to function as an electrode that may be used to detect electrical activity within the a target. This conductive portion may be conductively coupled to spring member1106, which is also conductive. In this manner, spring member1106may conductively couple the conductive portion of upper light guide portion1102with circuitry included on PCB1110. The circuitry may be configured to process the electrical activity detected by the electrode implemented by the conductive upper light guide portion1102in any of the ways described herein.

In some alternative example, both upper and lower light guide portions1102and1104are made out of the conductive material.

As shown, spring member1106comprises a coil spring positioned around an external surface of lower light guide portion1104. A proximal end of spring member1106pushes against PCB1110(or any other suitable support structure), while the distal end of spring member1106pushes against flange1108. Flange1108may be any suitable structure (e.g., a ring) attached to or protruding from upper light guide portion1102and/or lower light guide portion1104. By pressing against flange1108, spring member1106pushes the distal end of upper light guide portion1102away from upper surface1006of housing1002(shown inFIG.10). In this manner, the distal end of upper light guide portion1102may be biased away from upper surface1006of housing1002and toward the user's body.

In some examples, the multimodal measurement systems described herein may further include a processing unit configured to perform one or more operations based on photon arrival times detected by the detectors described herein and the electrical activity detected by the electrodes described herein. For example,FIGS.12A-12Bshow illustrative configurations1200-1and1200-2of an exemplary multimodal measurement system1202in accordance with the principles described herein.

Multimodal measurement system1202may be an implementation of multimodal measurement system600and, as shown, includes the wearable assembly602, light sources604, detectors606, and electrodes608described in connection withFIG.6.

In configuration1200-1, a processing unit1204is also included in wearable assembly602. In configuration1200-2, processing unit1204is not included in wearable assembly602(i.e., processing unit1204is located external to wearable assembly602). Either configuration1200-1or1200-2may be used in accordance with the systems, circuits, and methods described herein.

In configuration1200-2, processing unit1204is not included in wearable assembly602. For example, processing unit1204may be included in a wearable device separate from wearable assembly602. To illustrate, processing unit1204may be included in a wearable device configured to be worn off the head (e.g., on a belt) while wearable assembly602is worn on the head. In these examples, one or more communication interfaces (e.g., cables, wireless interfaces, etc.) may be used to facilitate communication between wearable assembly602and the separate wearable device.

Additionally or alternatively, in configuration1200-2, processing unit1204may be remote from the user (i.e., not worn by the user). For example, processing unit1204may be implemented by a stand-alone computing device communicatively coupled to wearable assembly602by way of one or more communication interfaces (e.g., cables, wireless interfaces, etc.).

In some examples, processing unit1204may be distributed between multiple devices and/or multiple locations as may serve a particular implementation. Processing unit1204may be implemented by processor108, controller112, control circuit204, and/or any other suitable processing and/or computing device or circuit.

For example,FIG.13illustrates an exemplary implementation of processing unit1204in which processing unit1204includes a memory (storage facility)1302and a processor (processing facility)1304configured to be selectively and communicatively coupled to one another. In some examples, memory1302and processor1304may be distributed between multiple devices and/or multiple locations as may serve a particular implementation.

Memory1302may be implemented by any suitable non-transitory computer-readable medium and/or non-transitory processor-readable medium, such as any combination of non-volatile storage media and/or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g., a hard drive), ferroelectric random-access memory (“RAM”), and an optical disc. Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

Memory1302may maintain (e.g., store) executable data used by processor1304to perform one or more of the operations described herein. For example, memory1302may store instructions1306that may be executed by processor1304to perform any of the operations described herein. Instructions1306may be implemented by any suitable application, program (e.g., sound processing program), software, code, and/or other executable data instance. Memory1302may also maintain any data received, generated, managed, used, and/or transmitted by processor1304.

Processor1304may be configured to perform (e.g., execute instructions1306stored in memory1302to perform) various operations described herein. For example, processor1304may be configured to perform any of the operations described herein as being performed by processing unit1204.

Processing unit1204may be configured to generate optical measurement data (e.g., fNIRS data) based on the arrival times detected by detectors606and electrical measurement data (e.g., EEG data) based on the electrical activity detected by electrodes608. This may be performed in any suitable manner.

For example, processing unit1204may be configured to process the optical measurement data and the electrical measurement data in accordance with a data fusion heuristic to generate an estimate of cortical source activity. In some examples, this may be performed in real-time while detectors606are detecting the arrival times and electrodes608are detecting the electrical activity.

To illustrate, an exemplary data fusion heuristic that may be employed by processing unit1204with respect to fNIRS and EEG data will now be described. The operations described herein assimilate samples of each modality as they become available and update the current estimates of cortical source activity in real time.

For the observation equation of the EEG, a standard linear propagation model may be represented by the following equation.
vk=Lsk+nk(1)

In equation 1, vkis a vector of voltages collected in the EEG sensors at instant k, L is the so-called lead field matrix that describes the propagation of electrical activity generated by sources in the cortex to the sensors, and skis the amplitude of the current source density in different parts of the cortex at sample time k, and nkis a sensor noise vector. The lead field matrix can be precomputed based on a model of the head derived from magnetic resonance imaging (MRI) data or an established atlas. For the observation equation of fNIRS, the following linearized model may be used.
fk=Jak+mk(2)

In equation 2, fkis a sample of oxy and deoxy absorption, J=MS, factorizes into the product of the MBLL linear transformation and sensitivity matrix S, akis a vector of light absorption on each location of the source space, and mkis optical sensor noise. The matrices M and S can be precomputed. To link the light absorption signal akwith the cortical electrical activity skin a computationally tractable manner, the following convolution model may be used.
ak=Σi=0nhisk−i(3)

In equation 3, the hicoefficients represent a low-pass FIR filter. Utilizing this approach allows for a fusion of EEG and fNIRS data that provides a method to link delay and strength of activation between the two modalities.

The data fusion heuristic described herein addresses at least two problems: 1) estimation of the vector time series of source activation skfrom the time series of sensor data vkand fk, and 2) estimation of the filter coefficients hifrom the source time series skand ak.

To address the first problem, equation 3 is plugged into equation 2 yielding:

fk=J⁢(∑inhi⁢sk-i)+mkfk=J⁢h1⁢sk+(∑i=1nhi⁢sk-i)︸Sk-1+mk(4)

In equation 4, Sk−1is the low-pass filtered version of the source time series up to the k−1 sample and can be considered fixed and known at the moment of estimating the sksource vector. Equation 4 may be used to rewrite equations 1 and 2 in a more compact way as follows:

[υkfk]=[LJ]⁢sk+[0J]⁢Sk-1+[nkmk](5)

Equation 5 is an ill-posed system because there are many more unknown sources than sensors. Hence, it may be solved for skusing a penalized least squares algorithm. To solve the second problem, equation 3 is rewritten in matrix form as follows.
Ak=Sh  (6)

In equation 6, Ak=[ak, . . . , ak−N] is a segment of light absorption signal and S is an embedding of past s source electrical activity. Equation 6 may be solved using least squares linear regression or any other suitable technique.

FIGS.14-19illustrate embodiments of a wearable device1400that includes elements of the multimodal detection systems described herein. In particular, the wearable devices1400shown inFIGS.14-19include a plurality of modules1402, similar to any of the modules and module configurations described herein. For example, each module1402may include a light source, a plurality of detectors, and one or more electrodes. The wearable devices1400may each also include a controller (e.g., controller112) and a processor (e.g., processor108) and/or be communicatively connected to a controller and processor. In general, wearable device1400may be implemented by any suitable headgear and/or clothing article configured to be worn by a user. The headgear and/or clothing article may include batteries, cables, and/or other peripherals for the components of the multimodal measurement systems described herein.

FIG.14illustrates an embodiment of a wearable device1400in the form of a helmet with a handle1404. A cable1406extends from the wearable device1400for attachment to a battery or hub (with components such as a processor or the like).FIG.15illustrates another embodiment of a wearable device1400in the form of a helmet showing a back view.FIG.16illustrates a third embodiment of a wearable device1400in the form of a helmet with the cable1406leading to a wearable garment1408(such as a vest or partial vest) that can include a battery or a hub. Alternatively or additionally, the wearable device1400can include a crest1410or other protrusion for placement of the hub or battery.

FIG.17illustrates another embodiment of a wearable device1400in the form of a cap with a wearable garment1408in the form of a scarf that may contain or conceal a cable, battery, and/or hub.FIG.18illustrates additional embodiments of a wearable device1400in the form of a helmet with a one-piece scarf1408or two-piece scarf1408-1.FIG.19illustrates an embodiment of a wearable device1400that includes a hood1410and a beanie1412which contains the modules1402, as well as a wearable garment1408that may contain a battery or hub.

In some examples, a non-transitory computer-readable medium storing computer-readable instructions may be provided in accordance with the principles described herein. The instructions, when executed by a processor of a computing device, may direct the processor and/or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and/or transmitted using any of a variety of known computer-readable media.

A non-transitory computer-readable medium as referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that may be read and/or executed by a computing device (e.g., by a processor of a computing device). For example, a non-transitory computer-readable medium may include, but is not limited to, any combination of non-volatile storage media and/or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, a solid-state drive, a magnetic storage device (e.g. a hard disk, a floppy disk, magnetic tape, etc.), ferroelectric random-access memory (“RAM”), and an optical disc (e.g., a compact disc, a digital video disc, a Blu-ray disc, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

FIG.20illustrates an exemplary computing device2000that may be specifically configured to perform one or more of the processes described herein. Any of the systems, units, computing devices, and/or other components described herein may be implemented by computing device2000.

As shown inFIG.20, computing device2000may include a communication interface2002, a processor2004, a storage device2006, and an input/output (“I/O”) module2008communicatively connected one to another via a communication infrastructure2010. While an exemplary computing device2000is shown inFIG.20, the components illustrated inFIG.20are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing device2000shown inFIG.20will now be described in additional detail.

Communication interface2002may be configured to communicate with one or more computing devices. Examples of communication interface2002include, without limitation, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio/video connection, and any other suitable interface.

Processor2004generally represents any type or form of processing unit capable of processing data and/or interpreting, executing, and/or directing execution of one or more of the instructions, processes, and/or operations described herein. Processor2004may perform operations by executing computer-executable instructions2012(e.g., an application, software, code, and/or other executable data instance) stored in storage device2006.

Storage device2006may include one or more data storage media, devices, or configurations and may employ any type, form, and combination of data storage media and/or device. For example, storage device2006may include, but is not limited to, any combination of the non-volatile media and/or volatile media described herein. Electronic data, including data described herein, may be temporarily and/or permanently stored in storage device2006. For example, data representative of computer-executable instructions2012configured to direct processor2004to perform any of the operations described herein may be stored within storage device2006. In some examples, data may be arranged in one or more databases residing within storage device2006.

I/O module2008may include one or more I/O modules configured to receive user input and provide user output. I/O module2008may include any hardware, firmware, software, or combination thereof supportive of input and output capabilities. For example, I/O module2008may include hardware and/or software for capturing user input, including, but not limited to, a keyboard or keypad, a touchscreen component (e.g., touchscreen display), a receiver (e.g., an RF or infrared receiver), motion sensors, and/or one or more input buttons.

I/O module2008may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O module2008is configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.

FIG.21illustrates an exemplary method2100that may be performed by processing unit1204and/or any implementation thereof. WhileFIG.21illustrates exemplary operations according to one embodiment, other embodiments may omit, add to, reorder, and/or modify any of the operations shown inFIG.21. Each of the operations shown inFIG.21may be performed in any of the ways described herein.

At operation2102, a processing unit generates optical measurement data based on a plurality of arrival times for photons of light after the light is scattered by a target within a user, the arrival times detected by a plurality of detectors included in a wearable assembly configured to be worn by the user.

At operation2104, the processing unit generates electrical measurement data based on electrical activity of the target, the electrical activity detected by a plurality of electrodes included in the wearable assembly.

At operation2106, the processing unit processes the optical measurement data and the electrical measurement data in accordance with a data fusion heuristic to generate an estimate of cortical source activity.

An illustrative multimodal measurement system includes a wearable assembly configured to be worn by a user and comprising: a plurality of light sources each configured to emit light directed at a target within the user, a plurality of detectors configured to detect arrival times for photons of the light after the light is scattered by the target, and a plurality of electrodes configured to be external to the user and detect electrical activity of the target.

Another illustrative multimodal measurement system includes a wearable assembly configured to be worn by a user and comprising: a light source configured to emit light directed at a target within the user, a detector configured to detect arrival times for photons of the light after the light is scattered by the target, and an electrode configured to be external to the user and detect electrical activity of the target.

Another illustrative multimodal measurement system includes a headgear configured to be worn on a head of a user and having a plurality of slots; a first module configured to be located in a first slot of the plurality of slots and comprising: a first light source configured to emit light directed at a target within the head of the user, and a first set of detectors configured to detect arrival times for photons of the light emitted by the first light source; a second module configured to be located in a second slot of the plurality of slots and comprising: a second light source configured to emit light directed at the target within the head of the user, and a second set of detectors configured to detect arrival times for photons of the light emitted by the second light source; and a plurality of electrodes on one or more of the headgear, the first module, or the second module and configured to detect electrical activity of the target.

In the preceding description, various exemplary embodiments have been described with reference to the accompanying drawings. It will, however, be evident that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. The description and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.