# Towards Domain-Robust Activity Recognition using Textual Representations of Binary Sensor Events

Ali Ncibi

<figure><img src="image_1.png" /></figure>

To cite this version:

Ali Ncibi. Towards Domain-Robust Activity Recognition using Textual Representations of Binary Sensor Events. ICAART 2026 - 18th International Conference on Agents and Artificial Intelligence, Mar 2026, Marbella, Spain. hal-05401764

HAL Id: hal-05401764

https://hal.science/hal-05401764v1

Submitted on 6 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards Domain-Robust Activity Recognition using Textual Representations of Binary Sensor Events

Ali Ncibi¹*

¹PETRUS Team (Inria Saclay), DAVID Lab. (UVSQ), France
{first.lastname}@inria.fr

**Keywords:**

Human Activity Recognition (HAR), Activities of Daily Living (ADL), Binary Sensors, Cross-Domain Adaptation, Textualization of Sensor Events, Language-based Representations

**Abstract:**

Language-based representations have recently emerged as a promising approach for cross-domain Human Activity Recognition (HAR) in smart homes, where binary sensor streams are verbalized into natural-language descriptions and processed by pretrained encoders. However, prior work has typically fixed both the textualization scheme and the text embedding model, leaving open how linguistic design choices influence transferability. This paper presents a comprehensive factorial analysis of textualization and embedding strategies for language-based HAR. We systematically vary (i) how sensor event windows are expressed—across seven existing and novel sequential and summarized textualizations—and (ii) how they are embedded using lexical (TF-IDF), static (Word2Vec), and contextual (SBERT) encoders. Experiments on four public smart-home datasets under consistent in-domain and cross-domain transfer conditions reveal that textualization design, not encoder complexity, governs performance. Sequential, event-ordered sentences maximize in-domain accuracy, while single-sentence, schema-based summaries—such as the proposed Compound Sensor Summary (CSS)—generalize best across homes. Clause-level ablations further show that event descriptions drive recognition, whereas explicit timing information can reduce robustness by overfitting to home-specific schedules. Overall, our findings establish a reproducible framework for analyzing and designing language-based representations in HAR, demonstrating that linguistic structure—rather than deep contextualization—is the primary determinant of domain robustness in smart-home activity recognition.ª

ªReproducibility materials are available at: https://github.com/ylaxor/flowar-textual-representations/

## 1 INTRODUCTION

Recognizing Activities of Daily Living (ADL) in smart environments from streams of binary events (e.g., door openings, motion detections) is essential for health monitoring, assisted living, and context-aware services. Most systems follow a data-driven pipeline of temporal segmentation of the continuous streams of binary sensor events, feature extraction, and personalized classification (Ncibi et al., 2025). However, deployment in new homes remains difficult: labeled data are scarce, and variations in sensors, layouts, and activity vocabularies hinder transferability across domains.

Early work on cross-domain adaptation tackled these issues through ontologies and probabilistic pri-

ors (Cook, 2010; van Kasteren et al., 2010; Chiang and Hsu, 2012), later replaced by representation learning methods that project sensor distributions into shared latent spaces (Sanabria and Ye, 2020; Sanabria et al., 2021; Hamad et al., 2022). Recently, language has emerged as a powerful alignment medium: sensor activations are verbalized into natural-language descriptions so that pretrained language models can extract domain-agnostic representations. Building on the initial demonstration by (Bouchabou et al., 2021), (Thukral et al., 2025) introduced Textual Descriptions of Sensor Triggers (TDOST) with BERT embeddings, and (Civitarese et al., 2025; Dhekane and Ploetz, 2025) further extended this line of work to few- and zero-shot reasoning.

Departing from prompting-based approaches, our study advances this line of work within a prompt-free paradigm. By not depending on external APIs for large

a https://orcid.org/0009-0004-2605-4936
language model inference, we improve reproducibility, protect user privacy, and facilitate real-world deployment—critical considerations for many in-home HAR applications.

Yet in this prompt-free setting, prior work generally adopts a single, fixed textualization and embedding setup, leaving unanswered how different linguistic design and encoding choices affect performance and transferability. In particular, it remains unclear whether deep contextual encoders are necessary for structured, low-ambiguity sensor data, or whether simpler linguistic formulations suffice. This paper addresses this gap through a factorial analysis of textualization and embedding strategies for language-based Human Activity Recognition (HAR). We systematically vary (i) how sensor event windows are expressed—across seven sequential and summarized textualization paradigms—and (ii) how they are embedded using lexical (TF-IDF), static (Word2Vec), and contextual (SBERT) encoders. We evaluate each configuration under the same in-domain and cross-domain transfer settings on four public smart-home datasets, allowing us to disentangle the respective impacts of linguistic form and representational depth. Our main contributions are:

1. **Systematic analysis:** We present the first factorial study disentangling textualization and embedding effects on in-domain and transfer performance.

2. **Novel textualizations:** We introduce three new textualizations of binary sensor events for activity recognition—two single-sentence variants of TDOST (TDOST-Plain-Basic and TDOST-Plain-Temporal) and the *Compound Sensor Summary (CSS)* textualizer. This last formulation provides a structured, clause-based schema that supports ablation of linguistic components.

3. **Empirical insights:** Through a two-way ANOVA analysis and clause-level ablations, we assess which informational elements (e.g., events, rooms, time bounds) most affect recognition and transfer, showing that structural regularity and information composition outweigh encoder complexity.

Taken together, these findings establish a reproducible framework and provide insightful guidelines for analyzing and designing language-based representations for human activity recognition, thereby advancing domain-robust smart-home systems.

**Paper organization.** In support of these contributions, the rest of the paper is structured as follows. Section 2 surveys prior work on language-based representations for cross-domain activity recognition. Section 3 introduces the proposed factorial framework, in-

cluding the textualization and embedding axes, as well as the classification setup. Section 4 details the four smart-home datasets employed in our experiments. Section 5 specifies the evaluation dimensions and the experimental in-domain and cross-domain protocol. Section 6 reports quantitative results and ANOVA analyses for both in-domain and cross-domain transfer. Finally, Section 7 summarizes the key findings, and Section 8 discusses current limitations and outlines avenues for future work.

# 2 RELATED WORK

## 2.1 Textualizations of Sensor Windows

Approaches for representing windows of binary sensor events through language can be broadly categorized into sequential and summarized textualizations. Sequential formulations explicitly preserve the temporal order of individual sensor trigger events, whereas summarized formulations condense an entire window into a single descriptive sentence.

<figure><img src="image_3.png" /><figcaption>Figure 1: Examples of existing textualization methods of sensor event windows in activity recognition. Textualizations follow authors' definitions or approximations when original specs are incomplete (Thukral et al., 2025; Civitarese et al., 2025; Dhekane and Ploetz, 2025).</figcaption></figure>

A common strategy is to recast the sensor stream as a time-ordered sequence of sentences, where each event is verbalized independently so that temporal evolution and co-occurrence are made explicit. In (Bouchabou et al., 2021), the feasibility of such textual abstraction in HAR was demonstrated by encoding each sensor activation as a short symbol combining sensor name and state, yielding a linguistic sequence of events. Building on this idea, (Thukral et al., 2025) introduced the TDOST framework, in which every
sensor firing is rendered as a self-contained natural language sentence that encodes contextual, spatial, and temporal information. Template-based formulations such as “motion sensor in bedroom fired with value ON” promote cross-home consistency, while variants like TDOST-Temporal further incorporate relational timing cues (e.g., “five seconds later…”). Subsequent extensions, TDOST-LLM and TDOST-LLM+Temporal, introduce lexical variety and more natural phrasing via GPT-based paraphrasing. Collectively, this progression from rigid templates to LLM-enriched sentences constructs a linguistic interface between quantitative sensor streams and pretrained language model semantics, balancing structural regularity for stable embeddings with contextual richness to support generalization across smart-homes.

By contrast, summarized textualizations collapse an entire window into a single sentence (or short to medium length paragraph), often encoding temporal order only implicitly. In (Civitarese et al., 2025), textualization is framed as narrativization, making sequences of sensor activations intelligible to LLMs through human-like stories. Their textualization method turns a time-bounded sequence into a concise narrative from the resident perspective—for example: “During a 60-second period around 7:00 am, the subject is in the kitchen. They open the fridge door and close it after a few seconds, then move to the living room and sit on the couch.” Clauses correspond to inferred actions linked by temporal adverbs (“then,” “after a few seconds”), preserving qualitative temporal order. The resulting sentences are subject-centered and use the active voice, matching the kind of everyday activity descriptions that LLMs are trained on. In contrast, (Dhekane and Ploetz, 2025) treat sentence construction as a structured summarization problem optimized for semantic embedding rather than narrative fluency. Within the “Thou Shalt Not Prompt” framework, entire sensor windows are encoded using templated declarative sentences that enumerate contextual attributes such as start and end times, duration, dominant locations, and most active sensors. For instance: “The activity started at 16:41 PM and ended at 16:42 PM. It was performed for forty-two seconds near the desk in the workspace and TV room. The two most commonly fired sensors were motion sensor near desk and motion sensor in workspace.” This rigid linguistic format enforces comparability across datasets and supports zero-shot classification via cosine similarity with predefined and verbalized activity templates instead of generative reasoning.

Together, these two textualization paradigms define a conceptual continuum: sequential textualizations emphasize event-level contextualization and

temporal grounding, while summarized textualizations prioritize higher-level semantic condensation through narrative or structured abstraction. Both approaches illuminate distinct philosophies of how language renders sensor data interpretable for recognition and reasoning, while leaving open the question of what precisely should be verbalized—and whether natural language itself is essential for generalization, or if domain-specific symbolic “languages” might better capture problem semantics.

## 2.2 Text Embedding Strategies

Once sensor windows are verbalized, they can be converted into numerical form using a variety of text-embedding techniques originating from NLP. Depending on how sentence representations are generated, these methods fall into two main families: those that build sentence embeddings from word-level representations, and those that derive them directly through contextual sequence models.

Early approaches, such as the classic Bag-of-Words (BoW) model and its Term Frequency-Inverse Document Frequency (TF-IDF) variants, represent sentences as high-dimensional lexical vectors, where each dimension corresponds to a vocabulary token weighted by its frequency or informativeness. While simple and interpretable, the resulting representations are sparse, disregard token order, and fail to capture semantic relationships among related terms.

The introduction of dense neural embeddings, such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), addressed these limitations by learning continuous token vectors from co-occurrence or predictive objectives. Sentence-level embeddings could then be obtained through aggregation methods—averaging, sum pooling, or weighting schemes like Smooth Inverse Frequency (SIF) (Arora et al., 2017)—with principal component removal (Mu and Viswanath, 2018) improving isotropy. Yet, these methods yield static embeddings that remain context-independent, assigning a fixed vector to each token regardless of its "meaning".

The emergence of contextual language models brought about dynamic, context- and meaning-aware representations. ELMo (Peters et al., 2018) utilizes bidirectional Long Short-Term Memory (LSTM) networks, whereas BERT (Devlin et al., 2019) adopts a Transformer-based self-attention mechanism to generate both token- and sentence-level embeddings via a special [CLS] token. Subsequent advancements—such as Sentence-BERT (SBERT) (Reimers and Gurevych, 2019), SimCSE (Gao et al., 2021), and more recent contrastive text embedding models in-
cluding E5 (Wang et al., 2022), GTE (Li et al., 2023), and BGE (Chen et al., 2024)—leverage large-scale contrastive training on text pairs to learn high-quality sentence and passage embeddings with strong transfer performance on semantic similarity and retrieval benchmarks.

NLP has largely moved from count-based features to contextual and semantic embeddings that capture token- and sentence-level meaning. In HAR, however, textualized sensor data are often highly structured and unambiguous, with little synonymy or linguistic ambiguity. This raises our second central question: Do contextual encoders actually improve transfer, or are simpler encoders sufficient for textualized sensor data?

# 3 METHODOLOGY

The previous strands of research—on textualization and embedding—define the foundation for our approach to language-based human activity recognition. In this section, we present a factorial framework that systematically varies both the textualization design and embedding strategy to reveal how their interaction governs both in-domain and cross-domain adaptation in HAR. Each combination is evaluated under identical transfer conditions to isolate the respective contributions of linguistic design and encoding choice.

## 3.1 Oracle Segmentation

Following prior work, we use oracle segmentation (aligned to ground truth annotations of activity occurrences) to control for segmentation error and isolate representation effects.

## 3.2 Textualization Axis

The textualization axis specifies how sensor event windows are expressed in natural language. All variants use the same metadata (type, location, object, state, timestamp); they only differ in wording.

<figure><img src="image_4.png" /><figcaption>Figure 2: Illustration of the window textualization phase.</figcaption></figure>

We evaluate seven representative textualizations covering both sequential and summarized paradigms:

* **Sequential textualizations:**

- TDOST-Basic (Thukral et al., 2025): baseline

sequential form with one sentence per event, referenced as TSB.

- TDOST-Temporal (Thukral et al., 2025): extends Basic by adding temporal cues, referenced as TST.

* **Summarized textualizations:**

- (Civitarese et al., 2025): concise narrative summary of key events, referenced as CTS.

- (Dhekane and Ploetz, 2025): templated summary ensuring consistency, referenced as DTS.

- TDOST-Plain-Basic: single-sentence variant of TDOST-Basic, collapsing all events into one structured description, and referenced as TPB.

- TDOST-Plain-Temporal: single-sentence variant of TDOST-Temporal, referenced as TPT.

- **Contextual Sentence Summary (CSS):** a new formulation designed to balance detail and compactness, described below.

<figure><img src="image_5.png" /><figcaption>Figure 3: Proposed examples of summarized textualizations: plain TDOST and the novel CSS, both expressing the same sensor-event window as a single structured sentence.</figcaption></figure>

The proposed Compound Sensor Summary (CSS) textualizer defines a compact, single-sentence verbalization scheme that encodes the structural and contextual composition of a sensor window. Concretely, it concatenates the following information about sensors within the window: (1) their types, (2) the rooms in which they are located, and (3) the event sequence they form (expressed as `object@room(type)=value`), along with global window attributes such as (4) the window duration, expressed in normalized textual form, and (5) its normalized time bounds (see Figure 3). This deterministic format prioritizes consistency and interpretability over narrative fluency, making it particularly suitable for clause-level analysis. Because each clause corresponds to a well-defined attribute, CSS enables controlled ablations that isolate the contribution of individual contextual dimensions—such as spatial diversity or temporal span—to downstream model performance. The formulation is
inherently extensible: additional features (e.g., sensor
density) can be seamlessly integrated as textual tokens
without modifying the underlying compositional syn-
tax. As a result, CSS supports scalable, linguistically
grounded evaluation of which informational factors
most effectively generalize across domains, subjects,
and sensor configurations.

Together, these variants establish controlled con-
ditions for analyzing how linguistic structure influ-
ences generalization.

## 3.3 Embedding Axis

The embedding axis specifies how textualized sensor
windows are subsequently transformed into numeri-
cal vectors before actual classification. To encompass
the principal representational paradigms, we adopt
three embedding families—lexical, static, and con-
textual—covering progressively richer linguistic ab-
stractions. When applicable, a common tokenizer is
employed across models to ensure vocabulary align-
ment, while transformer-based models retain their de-
fault tokenization schemes.

<figure><img src="image_6.png" /><figcaption>Figure 4: Illustration of the window embedding phase.</figcaption></figure>

*   **TF-IDF** serves as the lexical baseline, encoding textualized windows as high-dimensional, sparse vectors reflecting token frequency and informa-tiveness. Each dimension corresponds to a uni-gram or bigram (ngram_range = (1, 2)) selected from a vocabulary capped at 20 000 features. TF-IDF is a simple, interpretable baseline that mea-sures token importance but ignores context. This method relies purely on co-occurrence patterns in the training corpus and requires no external pre-training, making it a transparent yet order-insensitive reference point for more sophisticated embeddings.

*   **Word2Vec** model represents the static embedding family, capturing distributional semantics through vector spaces trained on large corpora (Google News, 300 dimensions). Pre-training uses a skip-gram objective that predicts surrounding words given a target word, enabling the model to encode syntactic and semantic similarity based on word co-occurrence statistics. In our pipeline,

each window sentence is tokenized, its word vec-
tors retrieved from the pre-trained vocabulary, and
aggregated by mean pooling weighted by corpus-
specific TF-IDF scores (idf_power = 1.0). This
produces a single 300-D representation per sen-
tence, balancing general linguistic knowledge with
domain-specific term importance.

*   **Sentence-BERT (SBERT)** encoder represents the contextual embedding family, modeling sentence-level meaning through deep transformer representations. Specifically, the all-distilroberta-v1 variant—fine-tuned with a Siamese architecture on the Natural Language Inference (NLI) task—produces dense 768-dimensional embeddings that preserve semantic similarity between paraphrastic sentences. Unlike lexical or static encoders, SBERT dynamically contextualizes each token based on its surrounding words, capturing subtle differences in phrasing and compositional structure. Following prior work (Thukral et al., 2025; Dhekane and Ploetz, 2025), we use frozen SBERT (no fine-tuning) so results reflect embedding quality, not task adaptation.

This range of embedding strategies—from sparse
lexical statistics to contextualized neural embed-
dings—enables systematic examination of whether
cross-domain adaptation arises primarily from linguis-
tic structure or from embedding sophistication.

## 3.4 Classification Models

Two standardized classification heads are used to re-
flect structural differences between sequential and
summarized textualization schemes.

*   (a) Sequential heads. For TSB and TST (more generally, for sequential textualizations), we employ a sequence encoder (e.g., BiLSTM, padded to a fixed length) operating over the list of sentence embeddings, followed by a dense classification layer. This setup mirrors the original TDOST framework, accommodates variability in sequence lengths, and remains constant across all implied evaluations to ensure fair comparison.

*   (b) Summarized heads. For textualizations that yield a single sentence per window, namely CTS, DTS, TPB, TPT, and CSS, a classification probe is directly applied to the single sentence embedding (e.g., MLP with one hidden layer).

Model architectures are selected manually to en-
sure comparability across textualization types while
keeping computational cost manageable. Although a
fine-tuning stage could further improve classification
<figure><img src="image_7.png" /><figcaption>Figure 5: Illustration of the classification phase.</figcaption></figure>

performance, it is intentionally omitted in order to isolate the effects of textualization and encoding, rather than model optimization. Fine-tuning is particularly costly for sequential textualizations, which require recurrent sequence models such as the chosen bidirectional LSTMs. These models involve greater parameter complexity, longer training times, and increased sensitivity to hyperparameter settings—highlighting an inherent limitation of sequence-based designs. Accordingly, single-vector windows verbalized through summarized schemes (DTS, CTS, TPB, TPT, CSS) are modeled using a multilayer perceptron (MLP) with one hidden layer of 128 units, trained with Adam ($lr = 0.001$), early stopping after 15 epochs of no improvement, and a 20% validation split. Sequence windows (TSB, TST) employ a bidirectional LSTM with 64 hidden units, sequence padding/truncation to length 100, and masking enabled to ignore padded positions. Both architectures share identical optimization and early-stopping settings for methodological consistency. This design ensures comparability across all conditions while highlighting the inherent modeling complexity of sequential formulations.

## 3.5 Cross-Domain Transfer Setup

The study adopts a source-to-target transfer objective in which a classification model trained and validated on a labeled source home is transferred to an unlabeled target home, under the assumption of partial overlap in activity vocabularies (Chiang and Hsu, 2012; Thukral et al., 2025). Following related work, and to ensure methodological comparability, robust performance estimates for the classification models are obtained using three-fold cross-validation on the source domain. Performance is measured using balanced accuracy, which explicitly mitigates the effects of activity frequency imbalance (class imbalance) (Luque et al., 2019). After cross-validation, the selected classifier is retrained on the full source dataset and evaluated directly on the target domain (without any further fine-tuning). This standardized transfer protocol corresponds to an extreme zero-shot adaptation scenario, in which no labeled data from the target home are available during training. The same procedure is applied consistently across all experimental settings (see Figure 6).

# 4 DATASETS

Experiments are conducted on four publicly available smart-home datasets comprising annotated binary sensor events. These datasets are widely used benchmarks that capture heterogeneous characteristics (sensors, layouts, activities) both within and across projects. Specifically, ARUBA and MILAN are drawn from the CASAS Smart Home Project (Cook et al., 2013), while ORDONEZ-A and ORDONEZ-B originate from (Ordóñez, 2013). These two projects are representative of distinct families of datasets: CASAS homes rely almost exclusively on motion sensors, with a small number of door sensors, yielding dense, highly frequent, and relatively monotonous event streams; in contrast, the Ordonez datasets are dominated by more semantic, object-centric sensors (e.g., door contacts, pressure mats, faucet and toilet-flush sensors), producing sparser streams in which individual events are more informative but less frequent. This structural distinction, while implicit in prior work, has rarely been articulated explicitly, yet it visibly conditions the success or failure of methods in the representation and classification stages of the pipeline, which depend on statistics of event frequency and duration. By spanning both families of datasets, our evaluation explicitly probes whether the proposed textual representations and classifiers remain robust across heterogeneous sensing regimes, rather than overfitting to a single class of smart-home deployments.

<table><caption>Table 1: Overview of the evaluated smart home datasets.</caption><thead><tr><td>Dataset</td><td>Residents</td><td>Days</td><td>Sensors</td><td>Activities</td></tr></thead><tbody><tr><td>Aruba</td><td>1 (visits)</td><td>220</td><td>34</td><td>7</td></tr><tr><td>Milan</td><td>1 (visits)</td><td>83</td><td>31</td><td>11</td></tr><tr><td>Ordonez-A</td><td>1</td><td>14</td><td>12</td><td>10</td></tr><tr><td>Ordonez-B</td><td>1</td><td>22</td><td>12</td><td>11</td></tr></tbody></table>

The collected raw files are converted into a uniform activation-based representation, specifying the start time, end time, sensor ID, and activity ID, following (Ncibi et al., 2025). Periods without annotated activities are labeled as OTHER. Sensor metadata—including sensor type, location, and associated object—constitute an essential component of the textualization phase and are manually curated. Initially ambiguous sensors, whose mappings to specific objects or locations cannot be reliably determined, are excluded to ensure consistency, particularly in the Aruba and Milan datasets. Building on prior work by (Liciotti et al., 2020) and (Thukral et al., 2025), activity labels across all datasets are standardized into ten common categories—BATHE, ENTER_HOME, FOOD, GROOM, LEAVE_HOME, OTHER, RE-
<figure><img src="image_8.png" /><figcaption>Figure 6: Illustration of the cross-domain transfer setup.</figcaption></figure>

LAX, SLEEP, TAKE_MEDS, and WORK—to enable partial overlap across different homes. The FOOD category encompasses both meal preparation and eating activities. In order to ensure temporal consistency, all datasets are truncated to the first contiguous 14 annotated days, matching the duration of the shortest dataset (Ordonez-A).

# 5 EVALUATION DIMENSIONS

Our factorial framework jointly evaluates all textualization and encoding strategies from Section 3 in a 7 × 3 design, yielding 21 configurations tested across source-target dataset pairs. This setup systematically examines how linguistic structure and representational complexity shape generalization. By fixing segmentation, classifier, and evaluation protocols and metrics, it isolates main and interaction effects along these axes. To ensure realism, experiments are conducted in both in-domain and cross-domain settings, with and without the neutral activity OTHER: excluding it yields idealized benchmarks, whereas including it introduces natural variability from unstructured household behavior. Each configuration is therefore evaluated in paired (idealized/realistic) variants, enabling direct comparison of accuracy and robustness.

# 6 RESULTS AND ANALYSIS

We report balanced accuracy (BA) for the full 7 × 3 factorial of textualizations × encoders, decomposed by setting (in-domain vs. cross-domain) and realism (whether OTHER is modeled as an activity class). Unless noted otherwise, we aggregate scores as means across folds (in-domain) or across all source → tar-

get pairs (cross-domain). We run two-way ANOVAs (Type-II) with factors *textualizer* and *encoder* for each (setting, realism) condition (cell) and report conventional *F*, *p*, and partial *η*² values.

## 6.1 In-Domain Performance

**Key finding.** When training and testing on the same home, explicitly representing event order dominates: sequential textualizations achieve the highest BA, while encoder choice has negligible effect (see Figure 7).

* *Idealized (no other).* Under idealized conditions, the best found configurations are based on sequential textualizations: TSB+SBERT at 0.8738 BA and TSB+TFIDF at 0.8734; TST+WORD2VEC reaches 0.8733, while summarized variants trail.

* *Realistic (with other).* The best model remains sequential: TST+SBERT at 0.7837 BA, followed by TST+WORD2VEC 0.7805 and TSB+TFIDF 0.7734.

*Two-way ANOVA.* Under idealized conditions, the *textualizer* main effect is significant (*F*(6, 63) = 3.375, *p* = 0.006, *η*² = 0.243), while *encoder* is not (*F*(2, 63) = 1.035, *p* = 0.361); the interaction is not significant (*p* = 0.949). With *OTHER*, no factors are significant, consistent with higher variability.

## 6.2 Cross-Domain Performance

**Key finding.** Under domain shift (source home → unseen target home), summarized, schema-like sentences transfer better than long event sequences. Encoder sophistication has limited impact.

* *Idealized (no other).* The best configurations are summarized: TPB+TFIDF achieves 0.7527 BA.
<figure><img src="image_9.png" /><figcaption>Figure 7: In-Domain: balanced accuracy (means across sources) under idealized and realistic conditions.</figcaption></figure>

Close contenders include TPB+WORD2VEC and TPT+TFIDF. Sequential textualizations lag (e.g., TSB+SBERT 0.6607; TST+SBERT 0.6246).

* Realistic (with other). Absolute scores drop but the ranking persists. The best models are summarized: e.g., TPB+WORD2VEC at 0.5343, etc.

Two-way ANOVA. Idealized cross-domain shows a significant textualizer effect ($F(6, 231) = 4.072, p < 0.001, \eta_p^2 = 0.096$); encoder is not significant ($F = 0.036, p = 0.964$); no interaction ($p = 0.488$). Again, with OTHER, no factors are significant (e.g., textualizer $p = 0.310$).

<figure><img src="image_10.png" /><figcaption>Figure 8: Cross-Domain: balanced accuracy (means across source→target pairs) under idealized and realistic conditions.</figcaption></figure>

## 6.3 Interaction Textualizer×Encoder

**Key finding.** We find no reliable textualizer×encoder interaction in any condition (Table 2). Trends are descriptive only: SBERT *slightly* helps sequential models in-domain (e.g., TST+SBERT 0.7837 with OTHER), but the best cross-domain results consistently pair simple encoders (TF-IDF/WORD2VEC) with single-sentence textualizations. This supports the thesis that *linguistic schema design*, not encoder contextual depth per se, governs transfer.

<table><caption>Table 2: Two-way ANOVA (Type-II) per condition. A: In-Domain, No OTHER; B: In-Domain, With OTHER; C: Cross-Domain, No OTHER; D: Cross-Domain, With OTHER. We report *F* and *p* values only, for brevity.</caption><thead><tr><th>Condition</th><th>Effect</th><th>F</th><th>p</th></tr></thead><tbody><tr><td rowspan="3">A</td><td>Textualizer</td><td>3.375</td><td>0.006</td></tr><tr><td>Encoder</td><td>1.035</td><td>0.361</td></tr><tr><td>Interaction</td><td>0.423</td><td>0.949</td></tr><tr><td rowspan="3">B</td><td>Textualizer</td><td>1.418</td><td>0.222</td></tr><tr><td>Encoder</td><td>0.350</td><td>0.706</td></tr><tr><td>Interaction</td><td>0.355</td><td>0.974</td></tr><tr><td rowspan="3">C</td><td>Textualizer</td><td>4.072</td><td>0.001</td></tr><tr><td>Encoder</td><td>0.036</td><td>0.964</td></tr><tr><td>Interaction</td><td>0.960</td><td>0.488</td></tr><tr><td rowspan="3">D</td><td>Textualizer</td><td>1.195</td><td>0.310</td></tr><tr><td>Encoder</td><td>0.098</td><td>0.907</td></tr><tr><td>Interaction</td><td>0.655</td><td>0.794</td></tr></tbody></table>

## 6.4 Impact of Realism

Adding OTHER reduces BA across the board, with larger degradations under cross-domain transfer. The largest cross-domain drops are observed for summarized+lexical pairings: TPT+TFIDF decreases by 0.274, CSS+WORD2VEC by 0.261, and TPB+TFIDF by 0.256. In-domain degradations are milder: For instance, TST+TFIDF drops by 0.129 and TSB+SBERT by 0.106.

## 6.5 Which Textual Clauses Matter?

We ablate five clauses of the CSS summary (ROOMS, TYPES, EVENTS, DURATION, TIMEBOUNDS) to quantify their contribution to generalization. We fix the encoder to TF-IDF and evaluate across in-/cross-domain and ±OTHER. Figure 9 reports ΔBA relative to Full-CSS (with and without OTHER). Table 3 summarizes the best reduced models per condition, i.e., the ablated CSS variant with highest mean BA.

**Findings.** Removing EVENTS yields the largest losses in both in- and cross-domain settings (up to
<table><caption>Table 3: Best reduced CSS vs. Full–CSS (means across dataset pairs), per condition. A: In-Domain, No OTHER; B: In-Domain, With OTHER; C: Cross-Domain, No OTHER; D: Cross-Domain, With OTHER.</caption><thead><tr><td>Condition</td><td>Full–CSS</td><td>Best Reduced</td><td>Name</td></tr></thead><tbody><tr><td>A</td><td>0.786</td><td>0.845</td><td>Duration</td></tr><tr><td>B</td><td>0.728</td><td>0.768</td><td>Rooms</td></tr><tr><td>C</td><td>0.662</td><td>0.712</td><td>TimeBounds</td></tr><tr><td>D</td><td>0.524</td><td>0.519</td><td>Rooms</td></tr></tbody></table>

–0.16 BA in-domain with OTHER and –0.11 cross-domain; mean ΔBA reported in Figure 9). In contrast, dropping ROOMS or TYPES has only minor effects (within ±0.04 BA on average). Removing DURATION is mildly beneficial in-domain without OTHER (mean gain +0.06) but not elsewhere. TIMEBOUNDS stands out as the least transferable clause: its removal slightly improves cross-domain performance without OTHER (+0.05), yielding the best reduced model in that condition (Table 3). Under OTHER, small positive effects for ROOMS in-domain (+0.04) are offset by consistent degradations for TYPES and DURATION (–0.02 to –0.05). A one-way repeated-measures ANOVA over ablations (within-pair factor: clause) confirms a significant clause effect in all conditions (in-domain: $F = 8.34/13.79$, $p < 10^{-3}/< 10^{-4}$ for no/with OTHER; cross-domain: $F = 3.90/2.44$, $p = 0.004/0.046$), with stronger effects in-domain. Collectively, these results indicate that EVENTS tokens drive most in-domain gains, while TIMEBOUNDS contributes little to transfer and may even induce leakage or overfitting to schedule-specific patterns, making temporal abstraction more robust across homes.

<figure><img src="image_11.png" /><figcaption>Figure 9: CSS ablation results (TF-IDF encoder). Each bar shows the mean ΔBA vs. Full–CSS when removing one clause, across dataset pairs.</figcaption></figure>

**Takeaways.** (i) *In-domain*, making temporal order explicit is paramount; encoder choice is secondary.

(ii) *Cross-domain*, single-sentence, structured textualizations (TPB, TPT, CSS) paired with *simple encoders* (TF-IDF/WORD2VEC) generalize best, indicating that *schema stability* rather than contextual depth drives transfer. (iii) *OTHER* disproportionately stresses transfer, yet the superiority of summarized schemes persists. (iv) Among CSS clauses, *EVENTS* are most critical, while *TIMEBOUNDS* may hinder transfer by overfitting to home-specific schedules.

# 7 CONCLUSION

This work presented the first factorial study of language-based Human Activity Recognition (HAR) under cross-domain adaptation. By systematically varying textualization, embedding strategy, and data realism across four benchmark datasets of ADLs and binary sensors, we disentangled how linguistic structure and representational depth jointly influence generalization in smart-home environments. Our results show that textualization design—not encoder complexity—governs both in-domain accuracy and cross-domain robustness. Sequential, event-ordered textualizations such as the original TDOST yield the highest balanced accuracy in-domain, confirming the importance of explicit temporal order when source and target distributions align. Across domains, however, single-sentence, schema-based summaries—particularly TDOST-Plain and the proposed COMPOUND SENSOR SUMMARY (CSS)—achieve substantially better transfer, even with simple lexical or static encoders. This demonstrates that linguistic abstraction and structural regularity, rather than deep contextualization, enable domain-agnostic alignment. Two-way ANOVAs confirm these trends: textualization effects are significant in both in-domain ($p=0.006$) and cross-domain ($p=0.001$) conditions, while encoder and interaction effects remain non-significant. Representation gains thus stem from how sensor data are verbalized, not from model scale. The inclusion of the OTHER activity class introduces realistic, unstructured periods that reduce absolute performance but preserve the relative advantage of schema-based summaries, highlighting their robustness to noise and partial labeling. Contextual encoders such as SBERT provide only marginal benefits once linguistic form is well structured. Overall, this study establishes language as a robust intermediate medium for human-activity modeling, bridging symbolic sensor data and transferable embeddings. By decoupling linguistic design from encoder sophistication, it provides a principled foundation for future research on scalable and domain-robust HAR systems.
# 8 FUTURE DIRECTIONS

While this study demonstrates that linguistic representations can robustly support cross-domain transfer, several limitations suggest promising directions for future research. First, the current pipeline assumes manually curated sensor metadata (type, location, and object). Although this requirement is often manageable in controlled deployments, broader applicability—especially to public datasets with incomplete or noisy metadata—will require textualizations that tolerate uncertainty. Future work could leverage language-model completion to mitigate metadata dependency. Second, the analysis adopted an idealiized segmentation regime to isolate representation effects, assuming perfect alignment between sensor windows and activity boundaries (Thukral et al., 2025; Dhekane and Ploetz, 2025). While this yields an upper bound on achievable performance, it abstracts away the challenges of continuous inference. Extending evaluation to realistic segmentation scenarios—such as fixed-duration (Civitarese et al., 2025) or change-point-based windows (Aminikhanghahi and Cook, 2019; Bermejo et al., 2021)—will clarify how linguistic representations behave under deployment conditions. Third, although contextual encoders provided only modest quantitative gains, they remain promising for hybrid explainability frameworks. Because textualizations are inherently human-readable, they offer a direct interface for human-in-the-loop reasoning. Rather than relying on external prompting (Fiori et al., 2025), future work could explore whether structured, prompt-free pipelines can yield interpretable explanations and support interactive model inspection, allowing users to query or modify textual representations to refine system behavior. Fourth, while the factorial design disentangled textualization and embedding effects, it did not investigate fine-tuning strategies. Domain-adaptive objectives, contrastive learning, or multi-task setups may further illuminate the trade-offs between linguistic abstraction and contextual depth. Finally, the inclusion of the OTHER class highlights persistent domain realism challenges: unstructured and transitional household states degrade cross-domain accuracy more than in-domain performance, under scoring the need for improved out-of-distribution handling. These directions extend the proposed factorial framework toward a more comprehensive paradigm for interpretable, adaptive, and human-centered activity recognition, advancing language-based modeling from research prototype to deployable foundation for smart environments.

# ACKNOWLEDGMENT

This work is supported by a sponsorship agreement between La Poste Group, Inria, and the Inria Foundation. We thank Prof. Philippe Pucheral and Dr. Luc Bouganim for their valuable insights on related topics. AI tools, in particular ChatGPT, were used for text translation and shortening to meet formatting requirements.

# REFERENCES

* Aminikhanghahi, S. and Cook, D. J. (2019). Enhancing activity recognition using CPD-based activity segmentation. *Pervasive and Mobile Computing*, 53:75–89.

* Arora, S., Liang, Y., and Ma, T. (2017). A Simple but Tough-to-Beat Baseline for Sentence Embeddings. In *International Conference on Learning Representations*.

* Bermejo, U., Almeida, A., Bilbao-Jayo, A., and Azkune, G. (2021). Embedding-based real-time change point detection with application to activity segmentation in smart home time series data. *Expert Systems with Applications*, 185:115641.

* Bouchabou, D., Nguyen, S. M., Lohr, C., LeDuc, B., and Kanellos, I. (2021). Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes. *Electronics*, 10(20):2498.

* Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In *Findings of the Association for Computational Linguistics ACL 2024*, pages 2318–2335, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

* Chiang, Y.-t. and Hsu, J. Y.-j. (2012). Knowledge Transfer in Activity Recognition Using Sensor Profile. In *2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing*, pages 180–187.

* Civitarese, G., Fiori, M., Choudhary, P., and Bettini, C. (2025). Large Language Models Are Zero-Shot Recognizers for Activities of Daily Living. *ACM Trans. Intell. Syst. Technol.*, 16(4):78:1–78:32.

* Cook, D. J. (2010). Learning Setting-Generalized Activity Models for Smart Spaces. *IEEE intelligent systems*, 2010(99):1.

* Cook, D. J., Crandall, A. S., Thomas, B. L., and Krishnan, N. C. (2013). CASAS: A Smart Home in a Box. *Computer*, 46(7):10.1109/MC.2012.328.

* Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In *Burstein, J., Dorgan, C., and Solorio, T., editors, *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1* (Long and Short
Papers), pages 4171–4186, Minneapolis, Minnesota.
Association for Computational Linguistics.

Dhekane, S. G. and Ploetz, T. (2025). Thou Shalt Not Prompt: Zero-Shot Human Activity Recognition in Smart Homes via Language Modeling of Sensor Data & Activities.

Fiori, M., Civitarese, G., Choudhary, P., and Bettini, C. (2025). Leveraging Large Language Models for Explainable Activity Recognition in Smart Homes: A Critical Evaluation. *ACM Trans. Internet Things*.

Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Hamad, R. A., Yang, L., Woo, W. L., and Wei, B. (2022). Cross-Domain Activity Recognition Using Shared Representation in Sensor Data. *IEEE Sensors Journal*, 22(13):13273–13284.

Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. (2023). Towards General Text Embeddings with Multi-stage Contrastive Learning. *ArXiv*.

Liciotti, D., Bernardini, M., Romeo, L., and Frontoni, E. (2020). A sequential deep learning application for recognising human activities in smart homes. *Neurocomputing*, 396:501–513.

Luque, A., Carrasco, A., Martín, A., and de las Heras, A. (2019). The impact of class imbalance in classification performance metrics based on the binary confusion matrix. *Pattern Recognition*, 91:216–231.

Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. *Proceedings of Workshop at ICLR*, 2013.

Mu, J. and Viswanath, P. (2018). All-but-the-Top: Simple and Effective Postprocessing for Word Representations. In *International Conference on Learning Representations*.

Ncibi, A., Bouganim, L., and Pucheral, P. (2025). (Demo) FlowAR: A Framework for Data-Driven Development of Human Activity Recognition Systems using Binary Sensors. In *2025 21st International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT)*, pages 286–288.

Ordóñez, F. (2013). Activities of Daily Living (ADLs) Recognition Using Binary Sensors.

Pennington, J., Socher, R., and Manning, C. (2014). GloVe: Global Vectors for Word Representation. In Moschitti, A., Pang, B., and Daelemans, W., editors, *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.

Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep Contextualized Word Representations. In Walker, M., Ji, H., and Stent, A., editors, *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.

Sanabria, A. R. and Ye, J. (2020). Unsupervised domain adaptation for activity recognition across heterogeneous datasets. *Pervasive and Mobile Computing*, 64:101147.

Sanabria, A. R., Zambonelli, F., and Ye, J. (2021). Unsupervised Domain Adaptation in Activity Recognition: A GAN-Based Approach. *IEEE Access*, 9:19421–19438.

Thukral, M., Dhekane, S. G., Hiremath, S. K., Haresamudram, H., and Ploetz, T. (2025). Layout-Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST). *Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.*, 9(1):19:1–19:38.

van Kasteren, T. L. M., Englebienne, G., and Kröse, B. J. A. (2010). Transferring Knowledge of Activity Recognition across Sensor Networks. In Floréen, P., Krüger, A., and Spasojevic, M., editors, *Pervasive Computing*, pages 283–300, Berlin, Heidelberg. Springer.

Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training. *ArXiv*.