EximiusLabs/fusion-embedding-2-tactus-mat

Tactus Mat: open-vocabulary posture from a body pressure mat

A body pressure mat streams a 64x32 grid of numbers. Tactus Mat turns that grid into a point in a text embedding space, so you recognize what is happening on the mat by ranking sentences rather than by running a classifier trained on a fixed label set.

Try it on real held-out data in a notebook (no dataset download, no hardware): demo.ipynb · open in Colab. Type your own sentence and it ranks real mat windows against it.

This is the mat sensor profile of the Tactus tactile pack. Same architecture, same recipe, different sensor: Tactus reads a 32x32 pressure glove and names grasped objects, Tactus Mat reads a 64x32 body mat and names postures. Both project into the frozen text space of fusion-embedding-2, which is never modified.

What it does

from inference import TactusMatEmbedder

tm = TactusMatEmbedder.from_pretrained("EximiusLabs/fusion-embedding-2-tactus-mat",
                                       revision="v0.1-preview")
window = mat_frames                       # [F,64,32] or one [64,32] frame
for text, score in tm.rank(window, ["a person lying flat on their back",
                                    "a person lying on their left side",
                                    "a person in the fetal position",
                                    "an empty bed"]):
    print(f"{score:+.3f}  {text}")

Raw sensor counts go in with raw="fsa" (the training normalization is clip(raw/1000, 0, 1) for the Vista Medical FSA range).

Results

Held-out subjects (11, 12, 13 of PhysioNet PmatData), never seen in training. Top-1 carries one sample standard deviation across independent runs.

Metric	Result	Chance
17-position top-1, open-vocabulary, mean of 5 runs	0.957 ± 0.016	0.059
17-position top-3, mean of 5 runs	0.993 ± 0.010	0.176
17-position top-1, single frame (one run)	0.947	0.059
3-class (supine / right / left) via the 17-way ranking (one run)	0.987	0.333
3-class scored against unseen coarse phrases, all 7 runs	0.882	0.333

The last row is the open-vocabulary property in one number: the model was trained against 17 fine-grained posture phrases, then scored against three coarse phrases it never saw during training, with no retraining and no classifier.

That score is identical in every run, MAE-initialized and from scratch alike, because its errors are systematic rather than noise. 98.7% of them are the two positions where a subject lies on their back with one knee raised, which the model ranks as lying on that side. Excluding those two positions, coarse accuracy is 0.987. Whether a supine subject with the right knee up should answer to "a person lying on their right side" is a question about the dataset's label taxonomy, not about the model.

Published supervised baselines on this dataset report 99.6% on the 3-class task (ResNet-18, leave-one-subject-out) and 82.7% in the dataset paper. Those are closed-set classifiers: they answer three fixed labels and cannot answer a new phrase.

Real held-out frames with their ranked queries

Selection policy: for each posture shown, the held-out frame with the largest margin between the correct query and its nearest competitor.

Temporal queries over a session

Because the mat streams at 1 Hz, posture becomes a time series and questions like "when did they last move?" are answerable directly. The night shown is a disclosed composition: real recordings from one held-out subject, sessions concatenated in a fixed order; per-frame agreement with the recorded posture is 99%.

Model

Field	Value
Input	[F,64,32] pressure frames scaled to [0,1] (8 frames at training time)
Output	2048-d L2-normalized vector in the fusion-embedding canonical text space
Trained parameters	16.2M (13.5M ResNet-18-width trunk, 2.6M projector)
Frozen	the entire 2B language side; existing embeddings are unchanged
Training	4000 steps, AdamW, cosine schedule, contrastive against 17 posture phrases
Cost	about half an hour on one A10G

Masked-autoencoder initialization shows no clear gain on this sensor: 0.957 ± 0.016 over five MAE-initialized runs against 0.945 ± 0.007 over three from-scratch runs (+1.2 points, Welch t = 1.4, p = 0.22, under five test windows). The +6.6-point same-sensor pretraining gain reported for the glove profile does not reproduce here, most likely because this task saturates near 0.96 on 20k frames from a single mat while the glove task sat far from its ceiling.

The released checkpoint is one MAE-initialized run scoring 0.961, near the five-run mean; the artifacts on this page come from it.

Adding your own sensor

This model exists to show that the expensive part is already done. A new pressure sensor does not need a new foundation model, it needs a small head trained against the same frozen text space: hours of compute, not a research program. If you build or sell pressure arrays (bed, seat, wheelchair, insole, industrial mat) and want your sensor to answer language queries, the recipe and code are open, and we are glad to help.

Data, license, and limits

Trained on PhysioNet PmatData (pmd 1.0.0) experiment I: 13 subjects, 17 in-bed positions, ~20k frames at 1 Hz on a Vista Medical FSA SoftFlex 2048 (32x64 sensels), ODC-By v1.0. Please cite Pouyan et al., "A pressure map dataset for posture and subject analytics," IEEE BHI 2017, along with PhysioNet.

Weights are released under ODC-By 1.0, matching the training data.

Limits worth knowing before you rely on this:

• Thirteen subjects, one mat, one lab. Transfer to a different mat geometry needs a short fine-tune; our own cross-sensor ablation on the glove profile predicts that pooling across sensor families does not work.
• The test split is 408 windows, so one window is 0.25 accuracy points. Small differences on this dataset are noise.
• Not a medical device. This is a research demonstration, not a diagnostic or monitoring product, and it must not be used to make clinical decisions.

Family

Model	Sense
fusion-embedding-2	text, image, video, audio
Tactus	touch, 32x32 pressure glove
Tactus Mat (this model)	touch, 64x32 body pressure mat
Tremor	inertial motion
Ember	thermal

The memory layer these plug into is Engram (pip install engram-robomem), which does language and temporal search over recorded sensor sessions.