Article

https://doi.org/10.1038/s41467-023-42655-yMulti-modal deformation and temperaturesensing for context-sensitive machines
Received: 11 August 2023
Accepted: 17 October 2023

1234567890():,;

1234567890():,;

Check for updates

Robert Baines 1,2, Fabio Zuliani2, Neil Chennouﬁ
Rebecca Kramer-Bottiglio 1 & Jamie Paik 2

2, Sagar Joshi2,

Owing to the remarkable properties of the somatosensory system, human skincompactly perceives myriad forms of physical stimuli with high precision.
Machines, conversely, are often equipped with sensory suites constituted ofdozens of unique sensors, each made for detecting limited stimuli. Emerging
high degree-of-freedom human-robot interfaces and soft robot applications
are delimited by the lack of simple, cohesive, and information-dense sensing
technologies. Stepping toward biological levels of proprioception, we present
a sensing technology capable of decoding omnidirectional bending, compression, stretch, binary changes in temperature, and combinations thereof.
This multi-modal deformation and temperature sensor harnesses chromaticity
and intensity of light as it travels through patterned elastomer doped withfunctional dyes. Deformations and temperature shifts augment the light
chromaticity and intensity, resulting in a one-to-one mapping between stimulus modes that are sequentially combined and the sensor output. We study
the working principle of the sensor via a comprehensive opto-thermomechanical assay, and ﬁnd that the information density provided by a single
sensing element permits deciphering rich and diverse human-robot and robotenvironmental interactions.

Multi-modal sensing—detecting and responding to a variety of physicalstimuli—is a hallmark of human perceptive capabilities, yet has provendifﬁcult to replicate in machines. Soft sensors have striven toward
human levels of perception by serving as synthetic analogs to biological nervous system receptors1–3. Work on soft sensors underscores
the challenge of diversifying what state variables can be artiﬁciallyperceived while simultaneously reducing sensing infrastructurecomplexity.
The most common state variable measured by soft sensors isdeformation. Numerous proposed soft deformation-sensing technologies exist; among these, a large proportion utilize conductive
hyperelastic composites that change in resistance or capacitance as afunction of strain4–8. With a single conductive sensor, it is difﬁcult todiscern multiple deformation modes. Arrangements of multiple sensing elements may also be used to expand the range of detectablemodes9–12, but can burden systems with additional mass, fabricational

complexity, and often reliance on computationally expensive datadriven models to interpret sensor outputs.
An emerging class of soft optical-based deformation sensors has
several advantages over conductive sensors, including resilience toelectromagnetic interference, and greater relative information densitythat can be gathered from light (i.e., intensity, spectral wavelength,polarization). Harnessing light intensity13 or intensity and wavelength
together14–17 has resulted in detection of many deformation modes
from the μm to the cm scale. Heterogenous sensing mechanisms
combining optics, ﬂuidics, and conductive material packaged within asingle sensor body have been deployed to discriminate several individual and combined deformation modes via a learned model18.
However, there is still no sensor architecture capable of distinguishingomnidirectional bending, stretch, compression, and combinations
thereof with a single sensing element. Omnidirectional bending, speciﬁcally, is a ubiquitous deformation mode. It occurs among natural

1

School of Engineering & Applied Science, Yale University, 9 Hillhouse Avenue, New Haven, CT 06520, USA. 2School of Engineering, Ecole Polytechniquee-mail: jamie.paik@epﬂ.ch
Fédérale de Lausanne, EPFL STI IGM RRL MED 1 2313 Station 9, Vaud 1025, Switzerland.

Nature Communications | (2023)14:7499

1

Articlehttps://doi.org/10.1038/s41467-023-42655-yorganisms and artiﬁcial systems, the latter including soft joysticks andsoft robotic manipulators19–21. Knowledge of bending angle anddirection is critical for effective control of such systems, but typicallyrequires many sensors22,23.
In order to step machines toward biological levels of adaptive
behavior, next-generation sensors must decode more than just typesof deformation. In human skin, mechanoreceptors can provide bothtemperature and deformation sensing. Temperature sensing past athreshold can trigger myriad processes, including perspiration, pain,
and shivering. Similarly, in emerging soft machines, temperatures pastcritical thresholds can serve as an impetus for material modulus
changes24,25, shape-morphing26,27, actuation28,29, and self-healing30,31.
Combined soft temperature and deformation sensors have been proposed but frequently rely on materials that are toxic or fail under cyclicloading, restricting their application space25–35. Optical sensing oftemperature and very small uni-axial stretches in tandem (<0.1%) has
been demonstrated in conventional optic ﬁbers by exploiting Brillouin
loss36,37; other technologies for combined temperature and deformation sensing similarly are usually viable under uni-axial strains38,39.
To expand the perceptive abilities of machines, we created
ChromoSense: a sensing architecture capable of distinguishing omnidirectional bending, compression, stretch, discrete changes intemperature, and valid sequentially applied combinations thereof. The
sensor comprises patterned elastomer doped with colorful dyes,
including a thermochromic species (changes color when heated past atransition temperature). Deforming the sensor alters the path length oflight through the colorful sections; frustrated total internal reﬂection
results in a light chromaticity and intensity measured by an embedded
miniature spectral meter. Temperature is detected as a function of theshift in spectral transmittance of the thermochromic species. Encoding
dense yet disparate sensing modalities within a single soft sensor via a
cohesive medium, ChromoSense provides a foundational advancetoward sensing for human–machine interfaces and robots.

Results
Sensor design
ChromoSense is a stretchable cylinder made from dyed sections of
optically transparent rubber, connected between rigid 3D-printed
interfaces (Fig. 1 and Supplementary Fig. 1). The top rigid interface
holds a white LED with a 120° transmission angle. The bottom rigid
interface houses a miniaturized spectral sensor (see Note M1 for morefabrication information). Light from the LED travels inside the beam,
entering three sections that experience total frustrated internal
reﬂection (TFIR):ba

Color sensor

Clear

Dyed thirds
Heat

LED

Cool
Thermochromic section

cStretchOmni-directional bending

Compression

Temperature

Fig. 1 | ChromoSense: multi-modal deformation and temperature sensing.
a ChromoSense operates by measuring light wavelength and intensity as it travels
through sections doped with functional dyes. Scale bar: 1 cm. b Generalized components of the ChromoSense architecture: an LED shines into a composite beam,
consisting of dyed thirds, a thermochromic section that change chromaticity as afunction of temperature, a clear section that additively mixes the light, and a

Nature Communications | (2023)14:7499dminiature color sensor. Scale bar: 1 cm. c The proposed architecture can discern
stretch, compression, omnidirectional bending, temperature, and combinations
thereof indicated by the connectivity of the dotted lines. d ChromoSense enables
myriad applications spanning wearables, human-robot interfaces, and soft stimuliresponsive machines.

2Articlehttps://doi.org/10.1038/s41467-023-42655-y

Bending that bisects principal directions results in an approximate
average of the two color contributions opposite the direction ofbending (see Supplementary Movie 1). Two factors contribute tobending-induced chromaticity shifts. First, light intensity increases
along the side of the sensor opposite the direction of bending42; thecolors on that side consequently contribute more to chromaticity
output. Second, bending alters the path lengths of the individual colorful sections in the sensor, which act as selective wavelength ﬁlters
according to Beer’s law (see Note M4 and Supplementary Fig. 4 for
information about the operational principle of bending).
The additive mixing of different colors of light caused by varying
the bending direction provides continuous resolution. Ellipses ofincreasing radius centered about the undeformed chromaticity value
demonstrate this phenomenon (see Supplementary Fig. 5). Consequently, both the direction and curvature of bending may be determined from chromaticity alone. Intensity measurements whenbending further reveal the mechanism of sensor operation (Fig. 2a,bottom row). The intensity output of the sensor in bending proves tobe fairly repeatable, boasting less than 10% spread from the average
trajectory over the entire range of curvatures and bending directions.
Bending toward the maximum tested curvature generally results in adecrease in intensity because light exceeds the critical angle required
for TFIR (see Notes M4 and M5).
Tension and compression (Fig. 2b, c, respectively) negligibly alter
chromaticity because the relative path length through all dyed sections stays the same. There is a decrease or increase in the intensity
during tension or compression, with sensitivities of 13.4 dB ϵ−1 and2.3 dB N−1, respectively, where ϵ is strain. There is only a 2% spread intension intensity output from the average trajectory, whereas thisvalue is a greater 14% for compression. We attribute the lower
repeatability of the sensor in compression to the different stablegeometric conﬁgurations it assumes under loading.
Together, intensity and chromaticity uniquely differentiate axial
deformations from pure bending modes. Cyclic stretch-and-return
tests veriﬁed ChromoSense to be highly stable over 500 cycles, as itSingle-mode stimuli
We subjected ChromoSense to bending in three principal directions(see Note M3 and Supplementary Fig. 3), deﬁned as midpoints of eachcolorful cylindrical third, as well as bisections of those directions, while
monitoring both the chromaticity visualized in the CIE 1931 color
space40 (note that x and y coordinates are not position, but are traditional notation for chromaticity coordinates, functions of the tristimulus values41) and intensity of the light. We deﬁne the intensity oflight as ∑RGB, the sum of individual R, G, and B components detectedby the miniature spectral meter in the sensor. Results show distinct
color changes corresponding to bending in principal directions andtheir bisections (Fig. 2a, top row). Namely, there is a chromaticity shift
primarily toward the color(s) opposite the direction of bending. The
black arrows projecting experimental data to the plot boundary
visualize the spectral saturation toward which chromatic shift occurs.

b

0.38

0.6

0.32
0.24

0.4

0.3

0.2

0.8

0.356

0.8

0.6

0.348
0.27 0.285

0.6

0.4

0
-0.2 00.2 0.4 0.6 0.8
x

-1
-2
0
0.01 0.02
0.03
Bending Curvature (1/mm)

Δ Σ RGBΔ Σ RGB000-0.2 00.2 0.4 0.6 0.8
x

1.5

-2

1

0.5
0

0.4
0.2

5 10 15 20
Tensile Strain (%)

250

0.2 0.4 0.6 0.8
x

3

x10

2

Fig. 2 | Single input mode characterization. For each subﬁgure, the top row shows
CIE 1931 plots tracking light chromaticity, and the bottom row shows intensity.
White points are experimental data and black arrows project applicable experimental data linearly to plot boundary, showing which regions of spectral saturation
the bend directions tend toward. a Bending, where the inset zooms in to the portion of the CIE plot indicated by the black square; the color codes for each trajectory correspond to those in the intensity plot below. b Stretch, where negligible

Nature Communications | (2023)14:7499

0.6

2

x10

-1

-3

0.34750.271 0.2725

0.4

3

x10

Temperature
0.8

0.34850-0.2 00.2 0.4 0.6 0.8x

Δ Σ RGB

3

1

d

0.2

0.2

0
-0.2 0Compression

1.2
Δ Σ RGB

0.8

cStretchy

Bending

y

y

a

y

1. Longitudinally parallel cylinder thirds doped with red, blue, andgreen pigments, respectively, and bonded together. Changing thelength of one of these sections relative to others, i.e., throughbending, shifts the chromaticity output. Pure extension or compression, respectively, decreases or increases the light intensity
output with negligible impact on chromaticity.
2. A section doped with thermochromic microcapsules in series withthe polychromatic section. Occupying the entire cross-section, it
does not contribute to a chromatic shift during deformations, butdoes only when the temperature exceeds the activation thresholdof the thermochromism (Supplementary Fig. 2).
3. A neat section after all doped sections that additively mixes thelight into a homogeneous output.
TFIR occurs because of the higher refractive index of the sensor
material compared to its surroundings. A sensor may be clad in lower
refractive index material to increase robustness in applications involving extensive environmental disturbances, otherwise no cladding isneeded since air has a lower refractive index than the core material(see Note M2).

x10

0.8
0.4

Tapp= 109°C
T0 = 23°C

0

0.2 0.4 0.6 0.8 1
Compressive Force (N)

0

0

10

20
30
Time (s)40change in chromaticity is observed. c Compression, where negligible change inchromaticity is observed. d Temperature shift, starting from room temperatureand heated with 109 °C air via forced convection until the thermochromic dyereaches a steady state (when the entire volume of thermochromic doped material
exceeds the transition temperature of 31 °C). Conﬁdence bars/clouds indicate one
standard deviation for ﬁve tests with the same sample.

3

Article
exhibited no drift in chromaticity (see Supplementary Fig. 6). Pull-tofailure experiments revealed negligible change in chromaticity until
failure at a tensile strain of 61% (see Note M10 and Supplementary Fig. 7).
Heating the sensor from 23 °C past the thermochromic transition
point of 31 °C alters the absorbance of the thermochromic species,
desaturating it from a rich yellow (Fig. 2d). Although the thermochromism in the sensor has a binary temperature response (it does notfurther shift in color at temperatures higher than just above its transition temperature), the relative heat ﬂux can be inferred through thederivative of the color shift over time as the thermochromism
approaches its steady-state heated absorbance (Note M6 and Supplementary Fig. 8). The distance of CIE value shift at steady-state heatedabsorbance is large enough to decouple sensor readings of compoundmodal inputs involving temperature and bending, as will be discussed
in the next section.

Multi-mode stimuli
To evaluate ChromoSense’s ability to discern multi-modal stimuli, we
performed combined loading with bending + heat, stretch + heat,compression + heat, bending + stretch, and bending + stretch + heat. All
stimuli were applied sequentially. For example, for bending + stretch,the sensor was ﬁrst bent, then subjected to stretch. In cases with theaddition of heat, the sensor was ﬁrst brought to steady-state heatedabsorbance, then deformations were applied. Results attest to ChromoSense’s ability to distinguish two and even three sequentially combined modes (Fig. 3; the ﬁgure shows data during the last applied modein a sequence).
Owing to the strong spectral shift in the thermochromic species,
the chromaticity output for any mode(s) combined with heating has
distinct trajectories on the CIE diagram that do not overlap withunheated ones. For example, bending + heat trajectories have nooverlap with those of bending only (Fig. 3a). Likewise, heat + stretch
and heat + compression are shifted in origin from the corresponding
unheated modes (Fig. 3b, c). The angle splay of trajectories corresponding to bending directions on the CIE diagram resembles those in
the unheated state, with the exception of the bisection of blue and red,which is skewed ~20° from the unheated trajectory, and the bisectionof red and green, which exhibits a nonlinearity. We attribute these
discrepancies to the change in the refractive index of the thermochromic species upon heating, which augments the critical angle
required for TFIR depending on the bending direction.
Combined bending at the maximum tested curvature of
0.033 mm−1 with continuous strain up to 5% closely resembles chromaticity values observed in pure bending of Fig. 2a, but exhibits further decreases in intensity due to the addition of stretch (Fig. 3d).
Bending + stretch + heat likewise elicits similar chromaticity to bending + heat, but with further decreases in intensity due to stretch(Fig. 3e). Importantly, since bending induces notable chromaticity andintensity changes at the same time, while stretching only inducesnotable intensity changes, it is possible to decouple what portion of
the sensor output arises from which sequentially applied mode.
Observation of the intensity value—different from the change in value
shown in Fig. 3—in tandem with chromaticity facilitates decoupling(see Supplementary Fig. 9).
Application of heat generally increases ChromoSense’s sensitivity,and has negligible impact on repeatability. For example, whereas purestretch exhibited an intensity sensitivity of 13.4 dB ϵ−1, stretch and heatyielded an elevated 15.2 dB ϵ−1. Higher relative sensitivity occurs whenheated because the absorbance decreases when the thermochromism
desaturates (Fig. 2d). The one exception is compression, which
demonstrates comparable intensity change with the addition of heat
due to the small deformation range.
Overall, for a given deformation, the addition of temperature may
be decoupled from the sensor’s output because no chromaticity

Nature Communications | (2023)14:7499

https://doi.org/10.1038/s41467-023-42655-y

trajectory overlap occurs between its heated and unheated modes. A
combination of intensity and chromaticity signals permits sequentiallydecoupling bending + stretch as well as differentiating all of the multimodal cases from one another.

Applications
Owing to its ability to robustly decipher multiple combined stimuli,
ChromoSense unlocks a variety of applications in wearables,
human–machine interfaces, and soft robotics. First, when placed on asoft upper body-assisting exosuit, a single sensor gives dense state
feedback that can be used to infer the 3D pose of the wearer (Fig. 4a).
Existing applications with soft exosuits rely on myriad sensors toestimate 3D pose, bogging down the suit with hardware and increasing
computational effort to interpret sensor readings43,44. ChromoSense
matches the compliance of the soft exosuit, and is capable of capturing
the high DoF associated with natural human movement, speciﬁcallythe combined continuous rotary and out-of-plane DoF at joints. For
instance, whereas elbow or knee joints only require uni-axial sensing toinfer pose, other joints like the hip and shoulder exhibit a broader
range of motion that drives the need for multi-DoF sensing solutions.
Two sensors—one on the hip and one on the shoulder, where
often complex coupled motions are observed—were used to track awearer’s motions. Under in-plane torso ﬂexion, the hip sensor undergoes a linear change in chromaticity. The chromaticity and intensity ofthe shoulder sensor remain relatively unchanged (Fig. 4b). Shoulder
rotation results in a quasi-elliptical trajectory—a unique chromaticity
for each shoulder conﬁguration during rotation. The intensity signal
drops signiﬁcantly compared to that when bending at the torso,
despite a similar displacement in CIE space of within 5%, indicating thecompound stretching/bending of the sensor when the shoulder
rotates (Fig. 4c and Note M7).
These baseline tests allowed us to develop threshold regions in
CIE space corresponding to when the torso was bent over and the arms
were extended in a grasping conﬁguration. Color space threshold
control was then used to trigger assistance lifting a heavy object(Fig. 4d, Supplementary Movie 2, and Note M7). Valuable insights
about 3D pose that would require many uni-axial strain sensors typically used on soft exosuits can be gleaned from the results. For
instance, plotting the proportion of threshold (capped at 1) reached asa function of time reveals a phase offset of 0.5 s between torso and arm
conﬁguration retractions when lifting the object, indicating the user’s
preferential 3D body poses when lifting. In addition, the hip sensors’
CIE space trajectory exhibits a nonlinearity, suggesting the subject
deviated from bending purely in-plane when picking up the object.
According to the shoulder sensors’ CIE space trajectory, upon grasping
the object at t = 2, the subject retracted and extended his arms along a
different paths, though passed through a nearly identical conﬁguration in the middle of the movement (seen as the intersection point in
CIE space). Overall, the sensor provides dynamic and simultaneous
biomechanical pose estimation with reduced hardware, underscoring
its utility as a wearable device.
Second, ChromoSense was used to give state estimates for realtime omnidirectional bending of a compliant 3-DoF interface (Supplementary Movie 3). In the case of haptic interfaces, it is often desired
to reduce complexity and footprint of sensory hardware to preserve
device bandwidth and geometry. Toward this end, we placed ChromoSense within a compliant origami-inspired user interface, reducing
the number of sensors required to estimate its surface orientationfrom at least three (on each leg) to one (Fig. 5a)45.
ChromoSense facilitated highly accurate, real-time simulation of adigital twin via a minimal neural network mapping chromaticity values
to surface orientations (Fig. 5b and Note M8). Predicted and actual
Euler angles were recorded when a user moved the interface in “outand-back” spiral trajectories of varying speeds and deformation
amplitudes. The ﬁrst case of higher amplitude, lower frequency spirals
4Articlehttps://doi.org/10.1038/s41467-023-42655-yb

Bending + Heat
0.26

0.226

0.8

0.2256

0.22

0.6

0.223

0.6

0.225

0.22 0.28

0.4

0.262

0.4

0.268

0.267 0.2677

0.4

0.2

0.2
0
-0.2 0
x10

3

0.2

0
-0.2 00.2 0.4 0.6 0.8
x0-0.2 00.2 0.4 0.6 0.8
x

2

3

0

0

-1

-2

Δ Σ RGB

-2

2
1.5

-4

No heat avg.
00.01 0.02
0.03
Bending Curvature (1/mm)

d
0.8

y

0.6

-6
0

No heat avg.
5 10 15 20 25
Tensile Strain (%)

e

Bending + Stretch
0.37
0.33

0.8

0.4

0

No heat avg.
0 0.2 0.4 0.6 0.8 1
Compressive Force (N)Bending + Stretch + Heat
0.26
0.2
0.22 0.28

0.6

0.24 0.3
0.2

0.4
0.2

0
-0.2 00-0.2 00.2 0.4 0.6 0.8
x

0.2 0.4 0.6 0.8
x0
Δ Σ RGB0
Δ Σ RGB

1

0.5

-3
-4

0.2 0.4 0.6 0.8
x

x10

x10

y

Δ Σ RGB

Compression + Heat

0.8

y

0.6

c

Stretch + Heat

Δ Σ RGB

0.8

y

y

a

-20
-40

-40
-80

-60

0

1
2
3
4
Tensile Strain (%)

5

Fig. 3 | Compound input mode characterization. For each subﬁgure, the top row
shows CIE 1931 plots tracking light chromaticity, and the bottom row shows
intensity. White points are experimental data and black arrows project experimental data linearly to plot boundary, showing which regions of spectral saturation
the bend directions tend toward. Tests with heat were heated with 109 °C air via
forced convection until they reached steady-state heated absorbance. Then, thedeformation mode was applied. Tests with heat are shifted from the analogous
unheated tests, which are included in the plots for comparison. No chromaticity
trajectory overlap occurs between a given heated and unheated deformation

Nature Communications | (2023)14:7499

0

No heat avg.
5
1
2
3
4
Tensile Strain (%)

mode. Chromaticity and intensity data, together, distinguish all multi-modal cases
from one another. Note that only data during the last mode in the application
sequence is shown. a Bending + Heat, with inset showing a zoomed-in of the portionof the CIE plot indicated by the black square. b Stretch + Heat, with inset illustrating
negligible change in chromaticity. c Compression + Heat, with inset again showing
negligible change in chromaticity. d Bending + Stretch. e Bending + Stretch + Heat.
Conﬁdence clouds/bars indicate one standard deviation from the mean for ﬁve
tests with the same sample.

5Articlehttps://doi.org/10.1038/s41467-023-42655-yabZoomed chromaticity plot

Shoulder Rotation
0.4

Sensor Anchoring
0.39
0.38
Δ Σ RGBy

Velcro Fastener

Side View

0.37

0

0.36

Stretch + Bend

-200

Shoulder
Hip

-400
0.5

1

1.5
2
2.5
Time, t (s)

c

t=0

0.35

t=2

0.24

3

0.26

0.28

0.3

x
Zoomed chromaticity plot

Torso Flexion

t=0

0.4

90°

0.39
t=3

0.38
y

SPA

Fabric Webbing

Δ Σ RGB

0.37

Sensor

0

0.36

Bend

-200

0.35

Shoulder
Hip

-400
1

2

3
4
Time, t (s)

5

0.24

6

0.26

0.28

0.3

x

d

Zoomed chromaticity plot
Activate
SupportHeavy
objectt=0

t=2

t = 6.5

t=0

0.41
0.4

Δt = 0.5s

0.95y

Proportion of Threshold

1

t=2

0.39 Activation

Regions
0.38

0.9

t=0

Shoulder
Hip
1

2

3

4

Time, t (s)

5

6

0.37

7

t=2
0.26 0.27 0.28 0.29

0.3

x

Fig. 4 | ChromoSense-enabled state estimation for a soft exosuit. a Torsoassisting exosuit, consisting of pouch-type SPAs placed ﬂat on the body, overlappedby fabric straps that help to hold the SPAs against the body and transmit the forces
to different regions around the torso. ChromoSenses are anchored to exosuit on
the shoulder and hip with Velcro, as to enable easy repositioning to other parts ofthe exosuit. b ChromoSense detects the multi-modal 3D deformation of combined

stretch and bending that occurs when the shoulder rotates, and gives in a unique
CIE space value for each position in the rotation. c Similarly, another ChromoSense
detects ﬂexion of the torso. d Creating threshold regions in CIE space, we provide
assistance to a user donning the exosuit when lifting a heavy object. Actuators
inﬂate for supporting force when both sensors indicate body parts are in a lifting
position with arms extended upward and outward, and torso bent forward.

shows strong agreement between Euler angles, with the root-meansquared error of 0.0516 (radians) for α (roll), 0.0966 for β (pitch), and
0.1052 for γ (yaw) (Fig. 5c). The second case of lower amplitude yethigher frequency oscillations also has an excellent agreement, with theroot-mean-squared error of 0.0499 (radians) for α, 0.0784 for β, and
0.0915 for γ (Fig. 5d). Due to the linear response of ChromoSense with
deformation, and its ability to accurately capture user input acrossdifferent oscillation frequencies and curvature magnitudes, it furnishes a simple and reliable means for user-in-the-loop tracking of
high-DoF mechanism kinematics.
Third, we employed ChromoSense in a variable stiffness soft
robotic manipulator to respond to environmental temperature changes
through move-and-hold operations to two prescribed opposite-handed

curvatures, denoted ± κt (Supplementary Movie 4 and Note M9). No
system to date has demonstrated combined multi-modal deformationand temperature sensing for closed-loop control of a soft robotic
manipulator with a single sensing element. In fact, the majority ofthermally activated soft robotic systems are open-loop46–49.
The manipulator consists of an antagonistic pneumatic actuatorpair adhered together by a thermosetting polymer (Fig. 6a). The
thermoset has a glass transition temperature region, Tg, spanning
35–40 °C, which roughly coincides with the color shift temperature of
the thermochromic dye in the sensor. When the manipulator is
exposed to temperatures at or exceeding Tg, the modulus of thethermoset drops 4 × , allowing deformation by the actuators. The
sensor was bonded along the outside of the thermoset-actuator

Nature Communications | (2023)14:7499

6

Articlehttps://doi.org/10.1038/s41467-023-42655-yab
z' y' x'zyx

Top View

y
x

β

2
t=0

-4

0

γ

-2

0
x (mm)

5 -1
0

3

6
9
Time, t (s)

12

α

6

Predicted
Actual6
9
12Time, t (s)

0

-1
0
1

-5

3

1

4

t=0

6
9
Time, t (s)

12

1.5

Predicted
Actual
3 4.5 6
7.5
Time, t (s)1.5

3 4.5 6
Time, t (s)

7.5

1.5

3 4.5 6
Time, t (s)

7.5

1
2
0
-2

-5

0

-1 0
1

-4
3

0

-1 0
β

4

0

0

-1
0
1

y (mm)α6y (mm)

d

1

γ

c

0
x (mm)

50-1 0

Fig. 5 | 3-DoF origami interface end-effector pose estimation. a As the 3-DoF
compliant origami user interface deforms, the xyz triad ascribed to its end-effector
shifts to x'y'z'. b The digital twin updates real-time with the estimated conﬁgurationof the interface. c, d Left: Projected reference x–y coordinate system view of theinterface end-effector center as the user performs a spiral motion. Right: Predictedversus actual Euler angles of the end-effector during the trajectory.

assembly to facilitate temperature and bending curvature sensing andto decouple it from pressure-induced surface strains.
We devised an autonomous control policy for the manipulator(state machine depicted in Fig. 6b). This move-and-hold sequence wasseeded with ± κt, empirically speciﬁed a priori via chromaticity values
at deformed states of interest. Temperature changes were induced via
forced air in an environmental chamber.
Chromatic trajectories during the move-and-hold sequence
underscore ChromoSense’s ability to effectively interpret temperatureand bending state variables simultaneously. Individually plottedchromaticity coordinates (x and y) as a function of time hone in on
more detail of when each step of the move-and-hold sequenceoccurred (Fig. 6c, d; note that subﬁgures c and d highlight key states
via contour color-coding that match those of subﬁgure b). The moveand-hold sequence starts with the manipulator at 22 °C (Fig. 6d1).
Environmental heating to 40 °C then induces a strong chromaticity
shift in the thermochromic species, indicated by the red line. ChromoSense registers when the temperature is above Tg, triggering
inﬂation of an actuator (Fig. 6d2). Consequently, chromaticity shifts
leftward on the green line as the actuator strives to attain κt. Uponreaching κt, pressure is held constant. As the environment cools,
shown by the blue line, ChromoSense registers when the temperaturedrops below Tg, triggering deﬂation of the actuator. Curvature ismaintained in the absence of pressure; however, the restoring springforce of the manipulator reduces its magnitude by 20%, and the corresponding chromaticity shift is evidenced by the cyan line (Fig. 6d3).
This concludes half of the move-and-hold sequence. Heating theenvironment beyond Tg again causes the manipulator to relax back to
its undeformed, heated conﬁguration along the black line. The ensuing
triggered inﬂation along the green line (Fig. 6d4), cooling along theblue line, and release along the cyan line (Fig. 6d5) concludes thesequence for the two speciﬁed curvatures of equal magnitude andopposite sign.

Discussion

Nature Communications | (2023)14:7499

ChromoSense harnesses parallel and series patterns of elastomer
doped with functional dyes to detect rich contact and non-contact
stimuli—all in a single sensor. These stimuli include stretch, compression, omnidirectional bending, discrete changes in temperature, andvalid sequentially applied combinations thereof, that may be distinguished via chromaticity and intensity changes. One ChromoSense
can replace arrays of conventional sensors for state feedback ofwearables, human interfaces, soft robots, and prospectively a vastarray of other high-DoF mechanisms. As an illustration of such versatility and utility, we presented three demonstrations.
First, a soft exosuit was outﬁtted with two ChromoSenses toestimate 3D pose, providing insight into wearer biomechanics.
ChromoSense matches not only the DoF of the exosuit, but its
compliance as well. We then utilized a novel color space threshold
control to trigger force-assisted object lifting. Results indicate
ChromoSense has signiﬁcant ramiﬁcations for low-footprint wearable technologies.
Second, an origami interface was equipped with one ChromoSense to estimate its orientation in 3D space, accurately updating a
digital twin in real time. We substantially reduce the hardware required
for state information updates, enabling seamless user interface
experiences. Moreover, this demonstration showcased how state-ofthe-art predictive models like neural networks can be applied to
ChromoSense’s output to obtain high-resolution estimates of bending
magnitude and direction.
Third, one ChromoSense was used to facilitate autonomous,
simultaneous thermal signal processing and curvature control of avariable-stiffness robotic continuum manipulator that would traditionally require deformation sensors and separate temperature sensors to achieve. The ability to decouple multiple modes of stimuli intandem elevated the information density, enabling the manipulator to
respond adaptively to the environment.
7

Articlehttps://doi.org/10.1038/s41467-023-42655-yab
Top View
Pneumatic Actuators

Wait
for

T > Tg0.4
yT > Tg5

Deflated
at fixed

κf

2

0.36

4

Hold
Inflation

0.34
0.26 0.28

d

κ

1

3

0.38

|κ|=|κt|

T < TgSensor0.42 Zoomed chromaticity plot
1

Inflateκt

Thermosetc|κ|<|κt|

Start

0.3
x

0.32 0.34

-κ
3

2

5

4

x and y

0.4
y

0.35
0.3
0.25
0

3

2
1

Cool to 22°C
Heat to 40°C

4

5

x
500

1000

1500

2000

2500
Time, t (s)

3000

3500

4000

Fig. 6 | Autonomous variable stiffness manipulator. a Image of the variable
stiffness manipulator with a top view schematic depicting the various functional
layers. Scale bar: 13 mm. b State diagram for the program controlling the autonomous variable stiffness manipulator. c Chromaticity diagram of sensor integrated
within manipulator during the autonomous move-and-hold sequence. Important

events are enumerated on the diagram. d Top: pictures of enumerated events from(c). Bottom: Corresponding x (black) and y (gray) chromaticity coordinates plotted
as a function of time, in the context of forced environmental heating or cooling,
indicated by red and blue sections, respectively.

Leveraging intrinsic optical properties of sensor host materials
allows us to not only capture rich and diverse stimuli, but also avoid
explicitly accounting for complicated properties like strain ratedependence, electromigration, and Mullins’ effect that can be issues in
conductive composite sensors. The ChromoSense design paradigm istherefore well-disposed to multiple scales and amenable to numerous
designs beyond those presented herein (see Note M2 for a discussion
about trade-offs of certain design variables). Although ChromoSense
has a limitation in terms of its ability to decouple simultaneously
applied stimuli, we suspect that there are several data processing
approaches that could be used to address this in future work (see Note
M10). As one extension to the present work, we foresee that temperature could possibly be detected as a continuous variable if using achemical species that exhibits a continuous spectral shift as a functionof temperature, like thermochromic liquid crystals. Overall, by packing
multi-modal sensing functionality into a single robust sensing element,
we take the next leap toward biological levels of proprioception.

ratio by mass, and the second employed urethane (smoothON Inc.,
ClearFlex 30) mixed in a 10:9.4 ratio by mass. The speciﬁc concentration of dyes was: R - 1drop/11g, B - 3 drops/11g, G - 7drops/11g (EPODEX,Transparent Red, Blue, and Green—Drop-In Farbstoff). In both cases,
polymer constituents were mixed together for 2 min, de-gassed for
3 min, and left to cure at 23 °C for 24 h. Next, we bonded the thirds
together by using a thin layer of the same polymer. Then, the ﬁnishedtri-color cylinder was placed into another mold and polymer dopedwith the thermochromic microcapsules (Adafruit, Thermochromic
Pigment–Yellow–10 g) was poured in to create a layer of 1 mm heightsection in series. Finally, an additional 50-mm-thick layer of polymer
without any dye in it was deposited on top of the thermochromic
section. The resulting sensor is 101 mm long and 10 mm in diameter.
The LED (OSLONSignal, LUW CRBP.01) and color sensor (Adafruit,
TCS34725) are constrained by parts 3D-printed from Acrylonitrile
butadiene styrene (ABS). The color sensor mounted on a signal processing board was press-ﬁt into the color sensor holder. The particulardimensions of these sensors were chosen based on length scale andthe applications, though many different geometries are possible. We
discuss scaling considerations in Notes M2 and M4.

Methods
Note M1: ChromoSense fabrication details
An exploded view detailing one speciﬁc sensor embodiment’s components is presented in Supplementary Fig. 1. Sensors were fabricatedﬁrst by pouring polymer into 3D-printed (Formlabs, Form 3+) molds tocast the cylindrical thirds, each being 50 mm long. We made two typesof sensors. The ﬁrst used Polydimethylsiloxane (PDMS) mixed in a 10:1
Nature Communications | (2023)14:7499

Note M2: ChromoSense design considerations
The speciﬁc ChromoSense embodiment of Supplementary Fig. 1 andsubsequently employed for several demonstrations had design parameters that were empirically determined. These parameters are not8Articlenecessarily optimal for the chosen geometry, so a more rigorous
parametric study must be conducted to identify optimal parameters.
In our case, basic physical principles combined with a quantitative
assay of sensor performance guided this parameter selection. Key
performance metrics included (1) intensity of light at the output,
quantiﬁed via measuring a proxy, the sum of RGB components fromthe color meter; (2) distinctness of chromatic responses to a givenmode, quantiﬁed by measuring the dissimilarity between modal
responses (i.e., for bending, the angle between CIE swept paths inprincipal directions); (3) sensitivity, quantiﬁed by the magnitude of
RGB ratio changes at maximum bending angle; (4) stiffness and stretch
at yield, determined through uni-axial quasi-static tension tests.
The theoretical maximum length of the entire ChromoSense
depends ﬁrstly on the luminous ﬂux output from a given LED. Longer
sensors will generally require LEDs omitting higher luminous ﬂux inorder to transmit light down the entire length, since light diminishes in
power quadratically per unit distance. Shorter sensors will therefore
have lesser ﬂux requirements. Optical properties of the sensor material
will further dictate the required power of an LED. Materials that exhibit
high absorbance, scattering, and reﬂectance will mandate higher
power. Similarly, the concentration of functional dyes within thematerial must be tuned to allow enough light through, while simultaneously serving as selective wavelength ﬁlters.
If any modes are to be associated with a unique sensor output
chromaticity in the chosen CIE embedding, (1) deformations must leadto unique path lengths through colorful sections, and (2) the spectral
shift exhibited by the thermochromic section must prevent deformations in heated and un-headed conditions from overlapping. The former is addressed via the geometric conﬁguration of the different
ChromoSense sections as parallel and series assemblies, as discussed
in the main text. The latter entails selecting an appropriate concentration and volumetric occupancy of the thermochromic species toallow sufﬁcient absorbance, as well as choosing a thermochromism
with spectrally distinct heated and unheated phases. There are
numerous commercially available thermochromic substances, spanning organic dyes to liquid crystals. They are available for a wide range
of operational temperatures from −100 to 200 °C50. Organic dyes often
come in the form of easily dispersible microcapsules, such as the ones
used in this paper, and exhibit a binary color change above a temperature threshold. Mixes of thermochromic dyes can exhibit several
color changes at different threshold temperatures. Liquid crystals
exhibit essentially continuous change in color as a function of temperature; a typical application of these is to assess heating distributionof electronic components. With such an abundance of thermochromic
compounds, creating ChromoSense units capable of higher resolution
temperature and deformation sensing is certainly feasible.
Lastly, as with conventional optical ﬁbers, the chosen material for
the ChromoSense core must have a higher refractive index than its
surroundings, or cladding, in order to produce total frustrated internal
reﬂection. Two different polymeric, stretchable materials—Sylgard 184(DOW Chemical) with a refractive index of 1.41, and ClearFlex 30(SmoothOn Inc.) with a refractive index of 1.46—were used to makesensors during explorations for the present paper. The refractive index
of air is 1. In applications where a sensor does not experience signiﬁcant contact with other surfaces, it sufﬁces to proceed without
additional material cladding. For related optical sensing architectures,
we refer the interested reader to a collection of pertinent materials,
their refractive indices, and Young’s moduli as in reference17.
Overall, a number of coupled physical parameters and their
combination dictate the performance of a ChromoSense: dimensionsof the sensor sections, optical and mechanical qualities of the materials, concentrations of functional dyes, and thermochromic capabilities of dyes. A wide design space exists, in terms of different
sensor architectures, that could even expand sensed deformation modes.

Nature Communications | (2023)14:7499

https://doi.org/10.1038/s41467-023-42655-y

Note M3: multi-DoF characterization setup and othercharacterization tests
We used a platform consisting of ﬁve degrees-of-freedom (DoF) to
characterize the chromaticity and intensity output of the ChromoSense in a variety of pure bending modes of different curvature anddirection of curvature (Supplementary Fig. 3). The platform consists of
two separate kinematic mechanisms whose end-effectors are connected through the ChromoSense. The ChromoSense is mounted on
the bottom to an X-Y gantry (FUYU, FSL40) and to the top to a Z-rollpitch system (FUYU, FSL40 connected to two Dynamixel, AX-12).
To prescribe sensor centerline curvature κ, length L, and directionof curvature ϕ for characterization, we developed representations foreach kinematic mechanism via homogeneous transformation matrices
arising from the modiﬁed DH-parameters51 below:

Link (PP)

αi−1

ai−1

di

θi

1

0

0

280-dx

0

2

π/2

85

120-dy

π/2

3

π/2

0

50

0

Link (PRR)

αi−1

ai−1

di

1

0

0

275  d z

0

2

π=2

0

120

θ2  π=2

3

π=2

0

0

θ3  π=2

4

0

50

0

0

5

π=2

0

−40

0

θi

Here, [dx, dy, dz] and [θ2, θ3] are the controlled kinematic parameters. The sensor was modeled as a centerline curve assumed to be of
constant curvature, determined by the displacement between the two
end-effectors. Kinematic parameters to achieve desired κ* and ϕ* werecomputed by minimizing:mind x ,d y ,d z ,θ2 ,θ322ðκ  κ * Þ + ðϕ  ϕ* Þ + DðT × PRRÞ + CðL  L* Þ

s:t : Lb ≤ d x , d y , d z , θ2 , θ3 ≤ Ub

2

ð1Þ

ð2Þ

Here, T is the tangent vector at the top of the sensor and M is thetangent vector of the grips (cross product tending to zero enforcesalignment with the grips), the * superscript denotes user-input goalvalues, and C=100 and D = 1000. Note all variables in the objective arefunctions of the kinematic parameters. Boundary conditions andlength are conditionally weighted to enforce solutions for purebending without additional stretch or compression of the centerline.
Minimization was performed with a genetic algorithm coded in
MATLAB. The resulting outputs were converted to the motor
commands and implemented on the setup to achieve programmed
deformations.
We conducted tension and compression experiments on thesensor using an Instron 3345 ﬁtted with a 1 kN load cell. All tests wereconducted at a loading rate of 40 mm/min. For sensor characterization
tests depicted in Figs. 2 and 3, we applied deformations considering afactor of safety on failure, so we could test the same specimen multiple
times and quantify repeatability in the operational range of the sensor.
Compression was applied up to just before the buckling force of thesensor. In uni-axial experiments, tension was applied up to 25% strain(the pull-to-failure results are discussed in Note M10). For combined
bending and stretch tests, we used the Instron machine outﬁtted with acustom ﬁxture to enforce a ﬁxed, maximum curvature while stretching9Articlehttps://doi.org/10.1038/s41467-023-42655-y(Supplementary Fig. 9a). We set the maximum axial strain to 5%. Notethat certain physically unrealizable combined modes of deformation
were not tested. These unrealizable combined modes include compression + bending, because the sensor cannot maintain a ﬁxed curvature with additional compression boundary conditions, as well ascompression + stretch, since both conditions act along the axis of thesensor and effectively cancel each other out. For tests involving heat,
we applied heat at 109 °C via forced convection until the chromaticity
value reached a steady state. Steady state occurs when the entire
volume of doped material exceeds the transition temperature (31 °C)of the thermochromic species.

Note M4: working principle of ChromoSense in bending
Two factors help explain chromaticity shifts that occur during bending: (1) ray path propagation concentration; (2) path length modulation of individual colorful sections in the sensor that act as selective
wavelength ﬁlters. Regarding the ﬁrst factor, it has been shown that inbent ﬁber optic cables, light rays concentrate at the side opposite to
the bending direction. Ray concentration results in an increase in lightintensity at that side and a decrease in intensity at the side toward the
direction of bending42. In the case of a bent ChromoSense, colors
opposite the direction of bending will have relatively greater lightintensity coming through them, amplifying their contribution to
overall chromaticity.
To explain the second factor, ChromoSense was modeled as a 2D
beam in pure bending, assuming homogeneous and linear materials.
The axial strain of the beam when under curvature is given as:ϵz =  κRsinðθÞ

ð3Þ

Here, κ is the radius of curvature, R is the distance from the neutral axis
to a point in the beam cross-section, and θ completes the polar coordinate system. This equation elucidates that axial strain varies linearly
as a function of distance from the neutral axis, reaching a minimum
opposite (180° in θ) of the direction of bending, and a maximum in the
direction of bending (0° in θ). For example, when bending directly
toward the intersection of the blue and green thirds, red stretches
maximally, and the majority of blue and green are compressed. The
result of these multi-color sectional stretches on chromaticity can beunderstood via Beer’s law52, written as:
A = ϵc‘ð4ÞHere, ϵ is the molar (decadic) absorption coefﬁcient, c is the concentration of solution, ℓ is the path length through a medium, and A isthe resulting absorbance. As ℓ increases, so does the absorbance of themedium. Yet absorbance is not uniform across all wavelengths; indeed,stretching monochromatic (RGB) beams, we validated that the output
chromaticity shifts toward the beams’ color (i.e., stretching a red
beam, the ratio of red increases in the output).
Speciﬁcally, we conducted tensile tests (Instron 3345 ﬁtted with a
1 kN load cell) on 50 mm-long and 10 mm diameter PDMS beams doped
a single color, red (1 drops/11 g polymer), green (3 drops/11 g polymer),or blue (2 drops/11 g polymer). We applied a strain of 10% at a rate of
100 mm/min. Results illustrate how chromaticity change is linear withdisplacement, and increases toward the color of the dyed beam (Supplementary Fig. 4, top). For example, the blue beam shifts towards blue
as it is stretched. The clear beam exhibits negligible change in chromaticity when stretched. Linearly ﬁt slopes of intensity change for red,green, blue, and clear beams were −907, −791, −552, and −504 units/
mm, respectively. Differences in slope values can be attributed to thedifferent concentrations of dye used for each beam and different
absorbance characteristics of the dye. Signal intensity decreases linearly and monotonically with displacement. Therefore, under thismodel, a color contributes maximally to the chromaticity of the light

Nature Communications | (2023)14:7499

output when the section corresponding to that color is opposite thedirection of bending because it acts as a wavelength-selective ﬁlter ofincreased ℓ and the colors opposite it decrease in ℓ. We contend thatpath length modulation impacting the light ﬁltering, as described by
Beer’s law, as well as ray path propagation concentration, together
explain the chromatic response in bending.
Based on the combined optical physics explanation for bending,we expect that the radius of the ChromoSense should scale in proportion to its sensitivity in bending, because more dyed material ispushed away from the neutral axis and will thus stretch more under
the same radius of curvature. By the same logic, decreasing theChromoSense’s radius will be possible only to a certain limit governed by color meter’s ability to still distinguish changes in chromaticity. For the current ChromoSense embodiment, the sensitivity
in bending depends on the direction of bending, inclusion of heat,and the regime of curvature. For example, when the blue section is
predominately stretched, there is a sensitivity of 3.97 dB mm−1. If thegreen section is predominately stretched, this value is 2.24 dB mm−1.
When averaged over the three principal directions and their bisections in the curvature regime of 0.02 to 0.03 mm, sensitivity comesout to 3.31 dB mm−1. In the case of bending + heat, this average is anelevated 5.72 dB mm−1. This variance in sensitivity is also accompanied by inconsistently monotonically decreasing behavior forcertain bending directions. Bending such that blue lies within thestretched region of cross-section results in a monotonicallydecreasing intensity versus curvature. On the other hand, bending
such that red lies within the stretched region of cross-section results
in an initial increase then decrease in intensity. We attribute these
phenomena to the facts that: (1) the concentrations and absorbanceof each color dye are different (Note M5 presents UV–vis characterization); (2) the LED may not be perfectly emitting uniform radiation;and (3) the refractive index of the materials varies due to different
chemical dopants, causing different wavelengths and quantities oflight to escape the sensor when bending based on the bending
direction.

Note M5: UV–vis spectra of dyed polymers
Normalized spectral transmittance of doped polymer at different
wavelengths of light was determined using a UV/Vis Spectrophotometer (LAMBDA, 365) on three different samples doped withred, green, and blue dyes (EPODEX, Transparent Red, Blue, and Green—Drop-In Farbstoff). Results attest to the highly selective ﬁltering
effect the dyes have on light passing through them (Supplementary
Fig. 2a). The change in color of the thermochromic dye was quantiﬁed
via transmittance as well; these curves show how heating past thetransition threshold allows more light in general, and proportionallymore light from 550 nm wavelengths and upwards to pass through(Supplementary Fig. 2b). Optically, a change from yellow to off-yellow
occurs when the dye is heated; such color changes in ﬂuoran-based
thermochromic compounds are triggered by a temperaturedependent proton transfer between the acidic developer and theﬂuoran53 (Supplementary Fig. 2c).

Note M6: thermal characterization experiments
We used a heat gun (Steinel HG 1910 E) 5 mm away from an ChromoSense and applied different temperatures to study the chromaticity
shift it underwent. Higher temperature resulted in a faster change inchromaticity, as quantiﬁed by the slope of the RBG ratio components(Supplementary Fig. 8). For example, Tapp = 169 °C resulted in fasterchromaticity change than heating at Tapp = 49 °C. The speed of color
change is consistent with the mechanism governing temperaturedependent color changes in ﬂuoran-based thermochromic compounds: the sooner the energy barrier for proton exchange is reached,
the sooner the color change occurs53. The slope of the color components can therefore be used as an indication of heat ﬂux, potentially
10Article
expanding the utility of the sensor to dynamic temperature sensing
scenarios.

Note M7: soft exosuit
The soft exosuit consists of donned fabric vest with spanning meshstraps that transfer force from soft pneumatic actuators (SPAs) on theshoulders and back to provide postural assistance. ChromoSenses
were mounted in a 3D-printed legs with Velcro fasteners on the bottoms to anchor them to the exosuit in various locations. The anchors
for the shoulder joint contain three Velcro fasteners, one on each leg.
Due to the location of the shoulder joint relative to the sensor placement, the sensor length increases upon rotation of the shoulder,
resulting in simultaneous stretch and bending. Torso ﬂexion causes
the length of the sensor on the hip to decrease, thus the sensor buckles
if all three legs are anchored. The buckling is an instability whose
direction we cannot control, so we decided to have the top of thesensor move freely without axial compressive forces by removing the
top Velcro fastener.
A pressure regulator (SMC ITV1011-21F1N) was triggered via PWM
signal generated from an Arduino to inﬂate or deﬂate the pouch
actuators depending on the state estimation provided by the ChromoSenses. The pressure line was supplied with 20 kPa. The authors
afﬁrm that human research participants provided informed consent
for publication of the images in Fig. 4.

Note M8: origami interface
The origami interface shown in this work is a parallel platform consisting of three legs. It is fabricated through layer-by-layer manufacturing and employs the combination of several functional layers.
Rigid parts made of glass ﬁber are linked together by a ﬂexible Kapton
layer. The distribution of ﬂexible layer “joints” allows the structure to
exhibit compliance at speciﬁc, programmed locations. We laser-cut
parts for each layer and heat-bonded them together with solid adhesive. For more information about the origami interface, please see ref.
45. The three-legged self-standing platform is retroﬁtted with the
ChromoSense between its end effector and base, reducing the needed
sensors to determine any movement from at least three (for each joint)
to just one in the center of the interface.
For digital twin demonstrations, we trained a two-layer neuralnetwork using MATLAB ﬁtnet to associate a given RGB ratio triplet
from the ChromoSense to the interface’s top surface orientation. The
neural network includes one hidden layer with a sigmoid activation
function, and one output layer with a rectiﬁed linear unit output
function. The input dimension is a 3 × 1 vector of RGB values, and theoutput is a 3 × 3 rotation matrix ﬂattened to a 9 × 1 vector. The training
was performed with an Aruco marker to determine the ground-truth
orientation of the surface. We used the trained neural network to map
RGB values to an orientation so as to update a digital twin of theinterface in real time. The graphics simulation was coded with
TKinter54 and uses online optimization to compute the inverse kinematics of the structure.
Owing to the sensor’s one-to-one mapping between any bending
deformation and a chromaticity value, the model facilitated excellent
matching between the digital twin and experiments. These experiments veriﬁed that a minimal data-driven model is an effective way tomap sensor outputs to desired units of deformation.

Note M9: autonomous variable stiffness manipulator
We created the variable stiffness manipulator by casting two pneumatic actuators from Dragon Skin 10 (SmoothOn Inc.), and bonding
them to muslin fabric strain-limiting layers. These two actuators werethen conjoined along their strain-limiting layers via a 1-mm-thickthermoset polymer. In particular, a 10:4 by mass ratio of Jeffamine
D400 (Huntsman International) and Epon 828 (DOW Chemicals) was
mixed thoroughly for 2 min, degassed for 10 min, and poured into a

Nature Communications | (2023)14:7499

https://doi.org/10.1038/s41467-023-42655-ymold placed into an oven at 38 °C for 12 h. One ChromoSense wasbonded to the x–y plane neutral axis of the manipulator with Silpoxy(SmoothOn Inc.).
The manipulator assembly was placed within an environmental
chamber outﬁtted with heating elements, and pressurized air forcooling. A digital thermocouple within the chamber provided groundtruth temperature readouts during the experiments. We used miniaturized pneumatic pressure regulators55 to inﬂate the actuators on the
manipulator.
In the state machine depicted in Fig. 6b, state transitions takeplace if temperature and curvature are detected to pass speciﬁed
thresholds. Threshold detections are implemented as inequalities or
equalities. Values used in the expressions are empirically determined
CIE coordinates associated to the actual variable. For instance, thestate transition requiring ∣κ∣ = ∣κt∣ indicates that the present curvature κof the sensor equals the target deformed curvature κt. We determineda chromaticity coordinate [x, y] for each of κ and κt in a calibration
phase prior to running tests. As another example, the state transition
T > Tg indicates that the temperature T of the manipulator exceeds theglass transition Tg of the variable stiffness material embedded within it.
T is estimated to be greater than Tg because the thermochromic portion reaches a steady-state heated absorbance, indicating the manipulator is above 31 °C. As with curvature detection, chromaticity
coordinates are determined at steady-state heated absorbance andunheated states, and used in threshold expressions in the statemachine.

Note M10: limitations of ChromoSense in this paper
The remainder of the methods section will discuss limitations of the
PDMS ChromoSense embodiment used for characterization in thepresent paper, and suggest possible avenues for addressing theselimitations. Under uni-axial tension, the ChromoSense has a lower
detection limit of 0.02 mm. A pull-to-failure test reveals a continuing decrease in intensity as a function of stretch, and maintenanceof negligible change in chromaticity (see Supplementary Fig. 7). The
stress in the sensor cross-section (calculated with respect to thereference area) exhibits non-linearly increasing behavior until failure, at around 61% strain, when there is a sudden decrease in stressand intensity as the sensor is split in two. Failure tends to occur
along the adjoining colorful sections or at the interface between thethermochromic portion and the clear portion. We found that
increasing the doping concentrations of the dyes embrittles the
ChromoSense, advancing the onset of failure, and increasing theforce response during deformation. One possible solution to
improve the elongation at failure is to adjoin sensor sections—
cylindrical thirds, thermochromic portion, and clear portion—using
integrated manufacturing processes such as 3D printing, rather
than individually casting and then bonding them. To combat thedyes’ embrittling effects, lower concentrations could be employed,
but the sensor geometry must be scaled appropriately to retain thedesired net ﬁltering effect.
During bending, the lower detection limit depends on the direction, and is, on average, 3 × 10−4 mm−1. There is a saturation limit
around 150° (the exact value depends on the speciﬁc direction ofbending and the predominately stretched and compressed color
portions) past which the expected trend in chromaticity shift stops.
Namely, past the saturation limit, the color content of the output signal
no longer increases toward those colors on the convex side of thesensor. We hypothesize this behavior arises because the critical angle
required for TFIR has been exceeded, and light can readily escape thesensor. One possible way to improve the maximum sensed bending
angle is to further increase the refractive index of the sensor relative toits surroundings through choice of other materials or claddings.
Another point of note is that the sensor’s baseline chromaticity isquite sensitive to dye concentrations. In practice, we found that slight11

Articlevariations in RGB and thermochromic dye concentrations ( ± 5% byweight of a single dye) can impact resting chromaticity x and y coordinates offsets as much as ± 0.02 in each coordinate. Precise balance
between dyes is needed in order to elicit a baseline chromaticity that iswell-centered within the CIE diagram and undergoes approximately
linear trajectories in CIE space during bending deformations.
An additional challenge with ChromoSense is that the timedependent response of the thermochromic shift makes complete
decoupling of temperature and deformation modes possible only atsteady-state heated absorbance. For example, if only half of thevolume of thermochromic dye transitioned and then a bending
deformation occurred, we could not intuitively decouple temperaturefrom deformation modes. One possible way to speed up the sensor’s
response to temperature is to reduce the volume of the thermochromic part and correspondingly increase its concentration,
decreasing its thermal mass. Relatedly, the binary response of thethermochromism prevents continuous temperature detection. ChromoSense could not distinguish between any temperature just above
the thermochromic transition point and one much higher, like 40 °C
and 90 °C. Consequently, the current combined temperature anddeformation-sensing capabilities may not be sufﬁcient for applications
that are fast-paced or that require continuous temperature
estimations.
As a last point of note, decoupling and estimating the relative
contribution of various stimuli to the sensor output is possible based on
knowledge of intensity, chromaticity, and their empirically determinedbounds. For instance, for sequentially combined bending + stretch,bending entails changes in chromaticity and intensity at the same time,
whereas stretch entails negligible changes in chromaticity, enabling us
to decouple the contributions to the output signal by analyzing changes
in both channels throughout applied deformations. However, ChromoSense—like any sensor—requires a model to map output values toﬁnite strains and curvatures. A variety of optical waveguide-based sensing architectures have spurred complementary approaches to estimate speciﬁc magnitudes of stimuli from a sensor’s output. These
approaches include both data-driven15,18,56,57 and physics-based22,58
models. ChromoSense exhibits nonlinearities when subject to certain
combined stimuli; a chosen model must therefore account for such
nonlinearities in order to be applicable over the entire operational
range of the sensor. One promising avenue would be to employ aminimal recurrent neural network. Since ChromoSense provides a oneto-one mapping between any applied stimulus and the output signal,
the recurrent time horizon of information could yield ﬁnite strains,
curvatures, and heat presence for sequentially combined modes.

Data availability
The origami interface workspace data generated in this study areavailable at the public repository https://github.com/the-faboratory/
ChromoSense (https://doi.org/10.5281/zenodo.7799503). All otherdata needed to evaluate the conclusions in the paper are present in thepaper and/or the Supplementary Information. Additional data related
to this paper are available from the authors upon request.

Code availability
Code can be found at the public repository https://github.com/thefaboratory/ChromoSense (https://doi.org/10.5281/zenodo.7799503)59.

References
1.
2.
3.

Rich, S. I., Wood, R. J. & Majidi, C. Untethered soft robotics. Nat.
Electron. 1, 102–112 (2018).
Chen, D. & Pei, Q. Electronic muscles and skins: a review of soft
sensors and actuators. Chem. Rev. 117, 11239–11268 (2017).
Zhao, S. & Zhu, R. Electronic skin with multifunction sensors basedon thermosensation. Adv. Mater. 29, 1606151 (2017).

Nature Communications | (2023)14:7499

https://doi.org/10.1038/s41467-023-42655-yChen, J. et al. Superelastic, sensitive, and low hysteresis ﬂexible
strain sensor based on wave-patterned liquid metal for humanactivity monitoring. ACS Appl. Mater. Interfaces 12,22200–22211 (2020).
5. O’Brien, B., Gisby, T. & Anderson, I. A. Stretch sensors for humanbody motion. In: Proceedings of the SPIE, 905618 (eds Bar-Cohen, Y.)(San Diego, 2014).
6. White, E. L., Yuen, M. C., Case, J. C. & Kramer, R. K. Low-cost, facile,and scalable manufacturing of capacitive sensors for soft systems.
Adv. Mater. Technol. 2, 1700072 (2017).
7. Frutiger, A. et al. Capacitive soft strain sensors via multicore-shellﬁber printing. Adv. Mater. 27, 2440–2446 (2015).
8. Kim, S. Y. et al. Sustainable manufacturing of sensors onto soft
systems using self-coagulating conductive Pickering emulsions.
Sci. Robot. 5, eaay3604 (2020).
9. Din, S., Xu, W., Cheng, L. K. & Dirven, S. A stretchable multimodal
sensor for soft robotic applications. IEEE Sens. J. 17,
5678–5686 (2017).
10. Truby, R. L. et al. Soft somatosensitive actuators via embedded 3D
printing. Adv. Mater. 30, 1706383 (2018).
11. Acer, M., Salerno, M., Agbeviade, K. & Paik, J. Development andcharacterization of silicone embedded distributed piezoelectric
sensors for contact detection. Smart Mater. Struct. 24,
075030 (2015).
12. Thuruthel, T. G., Shih, B., Laschi, C. & Tolley, M. T. Soft robot perception using embedded soft sensors and recurrent neural networks. Sci. Robot. 4, eaav1488 (2019).
13. Meerbeek, I. M. V., Sa, C. M. D. & Shepherd, R. F. Soft optoelectronic
sensory foams with proprioception. Sci. Robot. 8, eaau2489 (2018).
14. Guo, J. et al. Highly stretchable, strain sensing hydrogel opticalﬁbers. Adv. Mater. 28, 10244–10249 (2016).
15. Scharff, R. B. et al. Color-based proprioception of soft actuatorsinteracting with objects. IEEE/ASME Trans. Mechatron. 1, (2019).
16. Lin, X. & Wiertlewski, M. Sensing the frictional state of a robotic skinvia subtractive color mixing. IEEE Robot. Autom. Lett. 4,
2386–2392 (2019).
17. Bai, H. et al. Stretchable distributed ﬁber-optic sensors. Science
370, 848–852 (2020).
18. Kim, T. et al. Heterogeneous sensing in a multifunctional soft sensor
for human-robot interfaces. Sci. Robot. 5, eabc6878 (2020).
19. Yang, B. et al. Reprogrammable soft actuation and shape-shifting
via tensile jamming. Sci. Adv. 7, eabh2073 (2021).
20. Shih, B. et al. Custom soft robotic gripper sensor skins for haptic
object visualization. In 2017 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), 494–501 (IEEE, 2017).
21. Robertson, M. A. & Paik, J. New soft robots really suck: vacuumpowered systems empower diverse capabilities. Sci. Robot. 2,
eaan6357 (2017).
22. Sareh, S. et al. Macrobend optical sensing for pose measurement insoft robot arms. Smart Mater. Struct. 24, 125024 (2015).
23. Heiden, A. et al. 3D printing of resilient biogels for omnidirectionaland exteroceptive soft actuators. Sci. Robot. 7, eabk2119 (2022).
24. Schubert, B. E. & Floreano, D. Variable stiffness material based on
rigid low-melting-point-alloy microstructures embedded in softpoly(dimethylsiloxane) (PDMS). RSC Adv. 3, 24671–24679 (2013).
25. Buckner, T. L., Yuen, M. C., Kim, S. Y. & Kramer-Bottiglio, R.
Enhanced variable stiffness and variable stretchability enabled byphase-changing particulate additives. Adv. Funct. Mater. 29,
1903368 (2019).
26. Boley, J. W. et al. Shape-shifting structured lattices via multimaterial
4D printing. Proc. Natl. Acad. Sci. USA 116, 20856–20862 (2019).
27. Liu, K., Hacker, F. & Daraio, C. Robotic surfaces with reversible,
spatiotemporal control for shape morphing and object manipulation. Sci. Robot. 6, eabf5116 (2021).
4.

12

Article
28. Miriyev, A., Stack, K. & Lipson, H. Soft material for soft actuators.
Nat. Commun. 8, 596 (2017).
29. Huang, X. et al. Shape memory materials for electrically-powered
soft machines. J. Mater. Chem. B 8, 4539–4551 (2020).
30. Bilodeau, R. A. & Kramer, R. K. Self-healing and damage resiliencefor soft robotics: a review. Front. Robot. AI 4, 48 (2017).
31. Van Meerbeek, I. M. et al. Morphing metal and elastomer bicontinuous foams for reversible stiffness, shape memory, and self-healing
soft machines. Adv. Mater. 28, 2801–2806 (2016).
32. Yang, H. et al. Soft thermal sensor with mechanical adaptability.
Adva. Mater. 28, 9175–9181 (2016).
33. Hong, S. Y. et al. Stretchable active matrix temperature sensor arrayof polyaniline nanoﬁbers for electronic skin. Adv. Mater. 28,
930–935 (2016).
34. Xin, Y., Zhou, J. & Lubineau, G. A highly stretchable straininsensitive temperature sensor exploits the Seebeck effect innanoparticle-based printed circuits. J. Mater. Chem. A 7,
24493–24501 (2019).
35. Kim, S. Y. et al. Highly sensitive and multimodal all-carbon skinsensors capable of simultaneously detecting tactile and biologicalstimuli. Adv. Mater. 27, 4178- 4185 (2015).
36. Bao, X., Webb, D. J. & Jackson, D. A. Combined distributed temperature and strain sensor based on Brillouin loss in an optical ﬁber.
Opt. Lett. 19, 141 (1994).
37. Paixão, T., Araújo, F. & Antunes, P. Highly sensitive ﬁber optic
temperature and strain sensor based on an intrinsic Fabry-Perot
interferometer fabricated by a femtosecond laser. Opt. Lett. 44,4833 (2019).
38. Lichtenwalner, D. J., Hydrick, A. E. & Kingon, A. I. Flexible thin ﬁlm
temperature and strain sensor array utilizing a novel sensing concept. Sens. Actuators A Phys. 135, 593–597 (2007).
39. Liu, H. et al. A ﬂexible multimodal sensor that detects strain,humidity, temperature, and pressure with carbon black andreduced graphene oxide hierarchical composite on paper. ACS
Appl. Mater. Interfaces 11, 40613–40619 (2019).
40. Fairman, H. S., Brill, M. H. & Hemmendinger, H. How the CIE 1931
color-matching functions were derived from Wright-Guild data.
Color Res. Appl. 22, 11–23 (1997).
41. Hunt, R. & Pointer, M. Measuring Colour, fourth edition. (John Wiley& Sons Ltd, 2011).
42. Di, H. Sensing principle of ﬁber-optic curvature sensor. Opt. Laser
Technol. 62, 44–48 (2014).
43. Samper-Escudero, J. L., Contreras-González, A. F., Ferre, M., Sánchez-Urán, M. A. & Pont-Esteban, D. Efﬁcient multiaxial shouldermotion tracking based on ﬂexible resistive sensors applied to
exosuits. Soft Robot. 7, 370–385 (2020).
44. Kim, J. et al. Reducing the metabolic rate of walking and runningwith a versatile, portable exosuit. Science 365, 668–672 (2019).
45. Zuliani, F. & Paik, J. Variable stiffness origami interface. In IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS)(IEEE, 2021).
46. Baines, R., Freeman, S., Fish, F. & Kramer-Bottiglio, R. Variable
stiffness morphing limb for amphibious legged robots inspired bychelonian environmental adaptations. Bioinspir. Biomim. 15,025002 (2020).
47. Buckner, T. L. & Kramer-Bottiglio, R. Functional ﬁbers for robotic
fabrics. Multifunct. Mater. 1, 012001 (2018).
48. Wang, L. et al. Controllable and reversible tuning of material rigidity
for robot applications. Mater. Today 21, 563–576 (2018).
49. Blanc, L., Delchambre, A. & Lambert, P. Flexible medical devices:
review of controllable stiffness solutions. Actuators 6, 23(2017).
50. White, M. A. & LeBlanc, M. Thermochromism in commercial products. J. Chem. Educ. 76, 1201 (1999).

Nature Communications | (2023)14:7499

https://doi.org/10.1038/s41467-023-42655-y
51. Craig, J. J. Introduction to Robotics: Mechanics & Control (AddisonWesley Pub. Co., 1986).
52. Ball, D. Field Guide to Spectroscopy (ASPIE Press, 2006).
53. Azizian, F., Field, A. J., Heron, B. M. & Kilner, C. Intrinsically thermochromic ﬂuorans. Chem. Commun. 48, 750–752 (2012).
54. Lundh, F. An introduction to Tkinter. www.pythonware.com/library/tkinter/introduction/index.htm (1999).
55. Booth, J. W., Case, J. C., White, E. L., Shah, D. S. & Kramer-Bottiglio,
R. An addressable pneumatic regulator for distributed control ofsoft robots. In 2018 IEEE International Conference on Soft Robotics(RoboSoft), 25–30 (IEEE, Livorno, 2018).
56. Cao, D., Hu, J., Li, Y., Wang, S. & Liu, H. Polymer-based opticalwaveguide triaxial tactile sensing for 3-dimensional curved shell.
IEEE Robot. Autom. Lett. 7, 3443–3450 (2022).
57. Piacenza, P., Behrman, K., Schifferer, B., Kymissis, I. & Ciocarlie, M. A
sensorized multicurved robot ﬁnger with data-driven touch sensing
via overlapping light signals. IEEE/ASME Trans. Mechatron. 25,
2416–2427 (2020).
58. To, C., Hellebrekers, T., Jung, J., Yoon, S. J. & Park, Y.-L. A soft
optical waveguide coupled with ﬁber optics for dynamic pressure and strain sensing. IEEE Robot. Autom. Lett. 3, 3821–3827(2018).
59. Baines, R. et al. Multi-modal deformation and temperature sensing
for context-sensitive machines. ChromoSense https://doi.org/10.
5281/zenodo.7799503 (2023).

Acknowledgements
R.B., F.Z., N.C., S.J. and J.P. were supported by NCCR Robotics, aNational Centre of Competence in Research, funded by the Swiss
National Science Foundation (grant number 51NF40_185543). R.B.
and R.K.B. were also sponsored by the Ofﬁce of Naval Research underaward N00014-21-1-2417. Any opinions, ﬁndings, and conclusions or
recommendations expressed in this material are those of the authorsand do not necessarily reﬂect the views of the Ofﬁce of Naval
Research.

Author contributions
R.B. conceived of the concept and conducted fabrication and characterization of the sensors. F.Z., N.C. and R.B. implemented thedemonstration with the origami platform. N.C. created the digital twin
visualization. S.J. and R.B. conducted exosuit demos. R.B. did the variable stiffness manipulator demonstration. R.K.B. and J.P oversaw andguided the research. All authors contributed to writing the paper.

Competing interests
The authors declare no competing interests.

Additional information
Supplementary information The online version contains
supplementary material available athttps://doi.org/10.1038/s41467-023-42655-y.
Correspondence and requests for materials should be addressed toJamie Paik.
Peer review information Nature Communications thanks the anonymous reviewers for their contribution to the peer review of this work. A
peer review ﬁle is available.
Reprints and permissions information is available at
http://www.nature.com/reprints
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional afﬁliations.

13Articlehttps://doi.org/10.1038/s41467-023-42655-y

Open Access This article is licensed under a Creative Commons
Attribution 4.0 International License, which permits use, sharing,
adaptation, distribution and reproduction in any medium or format, aslong as you give appropriate credit to the original author(s) and thesource, provide a link to the Creative Commons license, and indicate ifchanges were made. The images or other third party material in thisarticle are included in the article’s Creative Commons license, unlessindicated otherwise in a credit line to the material. If material is notincluded in the article’s Creative Commons license and your intended
use is not permitted by statutory regulation or exceeds the permitteduse, you will need to obtain permission directly from the copyrightholder. To view a copy of this license, visit http://creativecommons.org/
licenses/by/4.0/.
© The Author(s) 2023

Nature Communications | (2023)14:7499

14