ReMAP: Neural Model Reprogramming with Network Inversion and Retrieval-Augmented Mapping for Adaptive Motion Forecasting
Authors: Sharmita Dey
Sarath Ravindran Nair
Venue: NeurIPS 2024 poster (2024)
Keywords: Model reprogramming
Patient mobility
Time series
Forecasting
Retrieval
Network inversion
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Mobility impairment caused by limb loss, aging, stroke, and other movement deficiencies is a significant challenge faced by millions of individuals worldwide. Advanced assistive technologies, such as prostheses and orthoses, have the potential to greatly improve the quality of life for such individuals. A critical component in the design of these technologies is the accurate forecasting of reference joint motion for impaired limbs, which is hindered by the scarcity of joint locomotion data available for these patients. To address this, we propose ReMAP, a novel model repurposing strategy that leverages deep learning's reprogramming property, incorporating network inversion principles and retrieval-augmented mapping. Our approach adapts models originally designed for able-bodied individuals to forecast joint motion in limb-impaired patients without altering model parameters. We demonstrate the efficacy of ReMAP through extensive empirical studies on data from below-knee amputated patients, showcasing significant improvements over traditional transfer learning and fine-tuning methods. These findings have significant implications for advancing assistive technology and mobility for patients with amputations, stroke, or aging.

ReMAP: Neural Model Reprogramming with Network Inversion and Retrieval-Augmented Mapping for Adaptive Motion Forecasting
Sharmita Dey', Sarath Ravindran Nair*
Abstract
Mobility impairment caused by limb loss, aging, stroke, and other movement deficiencies is a significant challenge faced by millions of individuals worldwide. Advanced assistive technologies, such as prostheses and orthoses, have the potential to greatly improve the quality life for such individuals. A critical component in the design of these technologies is the accurate forecasting of reference joint motion for impaired limbs, which is hindered by the scarcity of joint locomotion data available for these patients. To address this, we propose ReMAP, novel model repurposing strategy that leverages deep learning's 's reprogramming property, incorporating network inversion principles and approach adapts models originally designed for able-bodied individuals to forecast joint motion in limb-impann impaired patients without altering model parameters. We demonstrate the efficacy of ReMAP through extensive empirical studies on data from eelow-knee-challenee
mapping. Our
patients, showcasing significant improvements over traditional transfer learning and fine-tuning methods. These findings have signifi- cant implications for advancing assistive technology and mobility for patients with amputations, stroke, or aging.
Introduction
Physical impairment is life- life-altering event that affects millions of individuals worldwide, imposing substantial challenges on their mobility and daily activities. The disability of a limb can lead to significant physical and psychological limitations, impacting the individual's overall well-being and independence. In recent years, considerable efforts have been made to develop advanced assistive technologies to address these challenges and enhance the quality of life for impaired patients.
A crucial aspect of designing effective assistive technologies is the accurate prediction of reference joint motion for the impaired limb. Understanding the natural motion of the joints essential for the development of prosthetic devices that can restore the function of the impaired limb seamlessly. However, obtaining reliable data for impaired patients is complex task, their numbers and ability to perform diverse motion conditions are relatively limited compared to able-bodias individuals. This scarcity of data hinders the training of robust models specifically tailored the unique conditions of each individual. Moreover, each impairment is a unique and individualized event, leading to wide range of motion patterns and functional variations among such patients. Consequently, creating single generic model for those patients is not practical, it would not capture the individual variations
Sharmita Dey conceptualized and led the project; both authors shared equal responsibility for executing the research.
Sharmita Dey is with the University Medical Center Göttingen, Georg-August University Göttingen tharmita.. tharmeta.deyymmdd.. uni- uni-uoott.... g-gettinggen de Sarath Ravindran Nair is with the Georg-August University of Göttingen, Germany wwaaiinrramairdeniig..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Foundation module g(:0g = {0g. Ut)
Refurbish module h(: hh)
(pretrained and frozen)
Mobility- impaired data
Task 1 head
Shared layers
Xamp
amp
X corr
Task 2 head
Yamp
y yamp
Task 3 head
Able-bodied data
Template mapping
Used during training only
X y
Figure 1: Simplified architecture of the proposed ReMAP. The corrupt inputs X Xamp from the individuals with mobility challenges are mapped clean inputs Xoorr computed from able-bodied individuals, and the corrected inputs are used to produce the desired motion variables for the individuals with mobility challenges Damp using a frozen foundation module pretrained for able- bodied subjects.
and diverse motion conditions that arise from different types of impairments. In contrast, subject- specific modeling attempts cater to each patient's 's specific needs and characteristics. However, this approach faces significant challenges well. The limited data availability for individual patients poses a hurdle developing accurate models, leading to suboptimal performance and generalization issues.
individual input
Model input-outuut space
Kepp
0.0
-0.5
neighborhood
0.0
0.2
0.4
0.6
0.8
10
0.70
0.55
output
1.
0.11
8 0.11
0.5
0.2
Yeep
0.55
0.4
0.0
0.78
1.0
0.8
-0.5
0.0
0.2
0.6
Xcorr
W/
x
/Earg V
X sc)
Figure 2: Illustration of computation of the correction input Xcorr corresponding the k--th input individual. The able-bodied input X that produces the most
sample X of the similar output as that of the desired mobility-impain individual output y is searched in the input-output space of the trained able-bodied foundation module. Instead of searching based on single desired motion variable Vampp sequence of values {yamm .... Vamp .... -**m) (marked by the red region in the lower left curve) is used and the able-bodied input X corresponding the midpoint the sequence considered. Further, a neighborhood of radius is considered around X and the correction input X computed weighted sum of samples in this neighborhood with weights decreasing (linearly exponentially) with increasing distance from the center X

---

In this research, we introduce ReMAP, a novel approach for motion generation in impaired individuals using neural model reprogrammingg Our strategy addresses the challenges of predicting joint motion with limited data by repurposing well-trained gait prediction models developed on able-bodied data. ReMAP utilizes network inversion principles and rerrieval-augmenter mapping to adapt models for impaired individuals without requiring retraining or fine-tuning of the pretrained models. ReMAP comprises three key components: (1) a diverse foundation module derived from able-bodied data, (2) template mapping module to identify the most relevant inputs for learning the mapping, and (3) a refurbish module that learns the mapping once the desired inputs are identified. Specifically, we leverage network inversion techniques [ 2 3 to generate correction templates for mapping impaired inputs to able-bodied patterns and retrieval based methods 4 5 to identify
the most relevant imputs for learning this mapping.
Furthermore, the adoption of neural model reprogramming provides an additional advantage in terms of cost and resource efficiency. Fine-tuning subject-specific models can be computationally expensive, requiring modifications to considerable number of parameters in the model. In contrast, model reprogramming optimizes smaller set of parameters, reducing the computational overhead significantly. Additionally, this technique preserves the model's original functionality, making adaptable to various tasks without compromising its initial training.
We evaluate our method against baselines using transfer learning and conduct extensive ablations on various reprogramming techniques and architectures, Our quantitative results demonstrate the effectiveness of our approach, particularly in very low-data regimes, where outperforms all tested transfer learning and fine-tuning methods. To the best of our knowledge, this is the first work in the domain of motion regression using neural model reprogramming. Our findings have significant implications for the advancement of assistive technologies, offering the potential to improve mobility and quality of life for limb-impaired patients.
2 Related Work
Reprogramming of models. The concept of model reprogramming involves repurposing proficiently trained models for novel tasks through data-level manipulation alone [8]. This underscores the capability of deep models to handle diverse tasks without the need to alter any of their model parameters. The efficiency of this reprogramming approach has been validated in the context of image classification 8 9 time-series classification [10 11, spoken command recognition [12], GAN conditioning 13 out-of- -ff--iiiiiiiitoionn detection 14), antibody sequence filling [15] and fraud
detection (66. This research explores the potential of model reprogramming address the challenges of predicting joint motion in assistive technologies. By repurposing well-established gait prediction models trained on bbdeeb data, we seek to provide limb-impaired individuals with improved mobility solutions. This approach has the potential to transform the landscape of prosthetic development, offering higher quality of life for individuals facing mobility challenges.
limbiimpaired individuals, aiming to enhance the development of
Network inversion. Network inversion techniques have been instrumental in optimizing neural network inputs to achieve specified outputs. Initially proposed by Linden and Kindermann [1] this method utilizes gradient- baased optimization to iteratively refine inputs until the network outputs the desired result 2 31. This approach has been studied and applied in interpretability research, such as visualizing deep convolutional networks to find the input image that maximizes the activation of particular neuron or layer [17]. understanding deep image representations [18] 19 20). and image
synthesis (21) In our work, we leverage this concept to generate correction template for mapping inputs, given the characteristics of the desired output motion.
Retrieval-based methods. Retrieval-based methods rely on retrieving relevant information from large corpus in response to a query 4 5 6 77. These methods involve identifying and retrieving relevant documents passages using techniques like dense retrieval (2) and using this information to inform subsequent processes such as text generation or model training. A notable application of Generation (RAG), where retrieval provides context
retrieval- based methods is that enhances generative processes [23 24]. We apply retrieval-baat based methods to identify segments of impaired-limb motion dynamics, to learn neural reprogramming module.
Gait motion models. Gait prediction is a challenging problem due to the complex nature of human gait. In recent years, there has been growing interest in developing gait prediction models
3

---

using machine learning techniques. Most works focused on developing such models by training
explicit models to learn the input-output synergy, especially, for able-bodied subjects 25]
26 22
28
29
30 31. with only a few works on lower-limb-impaired individuals 32
33
Directly
applying gait prediction model trained for able-bodied subjects for motion prediction in impaired individuals suffered in performance due to the inherent differences between gait patterns of able- bodied individuals and iimb-impaired individuals [34]. This study presents an efficient strategy to adapt generic gait prediction model for predicting limb joint motion during various locomotion tasks for below-knee impaired individuals, including walking and stair ascent/descent, without requiring model fine-tuning or retraining. Our method integrates retrieval with network inversion techniques 3 mapping impaired inputs to able-bodied patterns to facilitate accurate motion prediction.
Datasets
Able-bodied dataset. To train the able-bodied motion model, we utilize a comprehensive public dataset [35] that includes kinematic data collected from wearable sensors. These sensors comprise IMUs that measure 3D angular velocities and linear accelerations, well as goniometers that capture sagittal knee and ankle positions under various motion conditions. The dataset includes recordings from ten individuals performing a range of locomotion activities, such as level ground walking, stair ascent, stair descent, and walking on inclined walkways. The dataset comprises approximately 6.56 million samples across ten individuals, averaging about 656,039 samples per individual.
Lower-limb impairment motion datasets. Motion data were collected from individuals with below-knee lower-limb impairmenns/amputation (36] using a 200 Hz camera-based motion capture system (Vicon Motion Systems Ltd., UK) equipped with 12 cameras. Retro-reflective markers were placed on their bony landmarks in the torso, pelvis, thigh, shank, and residual foot, with additional markers on the thigh and shank for 3D tracking. Various locomotion tasks were performed, including different walking speeds, stair ascent, and stair descent. Gait event detection software (Vicon Nexus) marked gait cycle boundaries based on marker positions and force thresholds. We processed the marker trajectories and computed joint angles and kinematic data using OpenSim 37.. an open-source musculoskeletal modeling platform. The user studies were approved by the Institutional Review Board (IRB) of the University Medical Center Göttingen, Germany, Participants were informed beforehand about the experimental procedure, potential risks, precaution measures, and data protection. Experiments were conducted after obtaining written consent from participants. All ethical protocols regarding information, instructions, and compensation were followed. Details are provided in the appendix|A.. A.5
Inputs and outputs. We utilized the temporal history of angular velocities from the shank and thigh segments and the angular position of the knee joint on both sides as model inputs.
X; = (6(thigh.r) ofthiggh.) o..hhank.))
of(thonkh)) @(knee.r) o(knee.)) R KXD
(I)
where K = 20 is the length of history and D = 6 is the number of input features. The model predicted the angular position the ankle joint on the impaired side. Given that direct measurement of ground truth outputs from lower-limb impaired individuals is not feasible due to limb disabilities or loss, we computed the desired ankle motion trajectories for each such individual's gait cycle based on similar speed gait cycles of subset of able-bodied subjects with comparable anthropometric features (mass, age, height).
Σ Eee SunSunhhrppmmttr
Ene Nopeed 3k
Namp
Santhr cpottreppmetricl
Wspeed
where Santhropometrie the set of subjects with similar anthropometry as the impaired individual (height: +5cm, weight: 4:5kg) and Napeed is the set of gait cycles where the able-bodied subjects in walked similar speeds (+0..mm/)) as that of the impaired individual. The reference timepoint & for computing matching output for the impaired individual was based on the phase of the gait cycle.

---

4 Method
Our objective is to utilize a well-trained foundation model that has learned a variety of motion scenarios from physically capable individuals to predict the walking patterns of a lower-limb impaired individual's 's disabled limbs, all without fine-tume tne-ttning or altering the model parameters. This involves reprogramming/correeiing the inputs acquired from the impaired individuals by refurbishing them so that the foundation model is able to produce the desired motion variables from the limb-impaired individuals' inputs. Our method integrates retrieval with network inversion techniques, efficiently mapping impaired-individum inputs to able-hodied patterns and facilitating accurate motion prediction. It comprises three components: a foundation module, versatile multi-task model developed from data sourced from numerous able-bodied subjects; a retrieval retrieval-augmentee augmented template mapping module, which computes the correction template for the queried impaired individual input; and a refurbish module, designed to map the queried impaired individual input to the computed correction template. The foundation module then predicts the desired motion of the disabled limb from the refurbished impaired individuals' inputs. Figure 1 summarizes this architecture. Below, we elaborate on each of the components.
4.1 Foundation module
We train a foundation multi-task module using data from able-bodied individuals (Kab, Vas), encom- passing the different motion conditions. A fundamental difficulty in the realm of multi-task learning is finding a balance between utilizing the common patterns among multiple tasks while still preserving the adaptability needed to address the unique characteristics of each individual task. Our strategy involves constructing a predictive model, denoted as g()), which comprises two distinct components: shared core Is with parameters 0g that are shared across all tasks, and subsequently, task-specific layers gt. with parameters Oz. The holistic forecast for a given input instance Xt, pertaining to the task t at the time point k, is represented as g( 0(xr, t; 0g = {0., U, 0.)) ge (g. (Xe.x: 0,); 0,). For the shared core 9s. we employ time convolutions [38). We deploy lightweight task-specific layers, Jt- on top of the shared core, characterized by a two-layer multi- lalli-layerrrreee perceptron (MLP) with rectified linear unit (ReLU) nonlincarities. We compare architectures with task-specific final layers to a model without such task-specific parameters. In this alternate architecture, the output features (x; of the shared core for any task are passed through a single prediction head to produce the output motion variable. This comparison helps evaluate the efficacy of task-specific adaptations versus unified approach in predicting motion variables across different tasks.
4.2 Retrieval-augmenter template mapping
Once the able-bodied model is trained, we use it to predict the motion variables for the lower-limb mobility-impaired subjects without retraining the model with the affected individual's data. To achieve this, we map the mobility-ippaired individual's s inputs Xomp corresponding to a desired output Yamp corrected input Xeory such that the model gives similar output the mobility- impaired individual's 's desired output for this corrected input (Fig. D. We identify two methods for computing such correction input X Xeorr. namely nearest neighbor search and network inversion.
Nearest neighbor search. A naive way of mapping the impaired individual's input to an able- bodied template is to use the able-bodied input Xab for which the model gives the desired impaired individual's 's output jlmp as the corrected input, that is,
x corr
arg min]] gk sampll
(3)
X
where the desired impaired individual's 's output damp is computed from those able-bodied individuals sharing similar anthropometry (such as height, mass, and age) as that of the impaired individual.
However, an inherent problem of this approach is the ambiguity in the corrected input when similar values of desired output Famp occur separated in time. This can lead to entirely different input values being mapped as correction inputs for similar input values of the limb-impaired individuals. For example, in the illustration in Fig. 2 consider we were searching for a Xcorr for yamp 0.6 at
t = 0.6. However, yamp = 0.6 also occurs around I. = 0.2 for which the correction input might be completely different from the one at t = 0.6.
5

---

One way to disentangle this problem would be to consider not single yamp value for computing the correction input, but a sequence of co-occurring values time which the desired output Vamp is the midpoint (red region in Fig. 2). 'hee algorithm now searches for sequence of 2m + able-bodied inputs that would produce the desired sequence of mobility-impaires individual outputs {yamp 1 ..., ...,, ...,Vump Vamp: .... yang ). This way, it becomes less probable that different able-bodied input
sequences correspond to the same mubility-impaired individual output sequences. For example, the desired output Vamp at 0.6 in Fig. falls in the region of decreasing values of Yampp whereas the other similar value at around t = 0.2 lies in the region of increasing Yamp values. These two output values, although similar, would thus have different able-bodied inputs associated with them for correction. The correction input X corr in this case can be computed as
m Σ [IF((x -
x corr
arg min
Nampll
X
In the previous approaches, we used a single value of able-bodied input as a correction template for the mobility-impaired individual. However, this approach may be prone noise and overfitting. To deal with this, we propose an extension of our approach which integrates multiple able-bodied input values to form a correction template for the mobility-impaired individual. This is achieved by defining a e-neighborhood (black ellipse in Fig. [ around the able-bodied input for which the model prediction is closest to the desired mobility-impaired individual output (such able-bodied input is obtained using either Eqn. or Eqn. 4.. The correction template is computed as weighted sum of the able-bodied inputs within the e-neighborhood. If X be the closest able-bodied input around which the e-neighborhood is defined, the correction template is given by
X-
Σ jEarg(I -2
Wy X ab-
corF
ab where w, is the weight associated with each of the able-bodied input X in the e-neighborhood.
We define two types of weighting linear and exponential with the weight wy of an able-bodied
input sample X decreasing linearly exponentially it moves away from the center X of the c-neighborhood. The weighting factor Wy is thus given by
X
Xaa
if weighting linear
W0
x
X
(6)
exp(s
if weighting exponential
where is scaling factor that determines how fast the weight decreases the sample X moves away from the center of the e-neighhorhood. Other types of weighting (for example, uniform weighting of samples in the neighborhood) are also possible, but are out of scope of this work.
For simplicity, we consider only n closest neighbors within the e-neighborhood for computing the correction template Xeorr. Since the inputs are normalized in [0, 1]. is set to 0.01.
Network inversion. One limitation of the nearest neighbor search method detailed above is the requirement to store the complete training data used for training the foundation module, which increases the memory requirements. Although one can devise clever ways to store less data without affecting the accuracy, we propose directly retrieving the correction input X corresponding to the aabilitt-iippiire individual's ''s desired output, Damp from the pretrained foundation module, For the desired mobility-impaired individual output, slmmp at each time point k. we generate a correction template X corr such that,
x corr = arg min [x( - sampll2 + A /
where the À is the regularization strength.
This approach draws inspiration from network-inversion techniques introduced by Linden and Kindermann [ which involves finding a relevant input that produces specific output from a neural network. This technique, leverages gradient-base optimization iteratively adjust the input until the desired output is achieved.

---

4.3 Refurbish module
We perform a data-level adaptation of the mobilltty-impiiree individual inputs by learning a mapping from the corrupted mobility-impairee individual data X amp (due to compensatory motion, asymmetric gait) to the correction template c computed from the able-bodied data using the techniques mentioned in the previous section. To achieve this, we train a lightweight refurbish module h()) characterized by a multi-layer perceptron with three hidden layers of 100 units each. Following three strategies are employed for training the refurbish module.
Correction- baaeed mapping. The refurbish module h(.) is trained map the mobility-impaired individual input X Xamp to the corresponding computed correction X by minimizing the MSE loss between the computed correction Xeorr and refurbish module predicted correction X ((Xampi ²n) The loss function is
Limput
Σ
||((X ammp' O%) -
(0)-XXooma Xeorr
(8)
Xamps X E.t Xiruin
where Xtroin the training dataset consisting of mobility-impaired individual inputs and corre- sponding correction templates. The correction template Xeorr may be computed using either nearest neighbor search or network inversion, detailed the previous section.
Target-based mapping. In this case, the input to map to is not explicitly determined, instead the refurbish module optimizes its parameters to produce the target directly, given the frozen foundation module. The refurbish module h(.) is trained to minimize the MSE loss between the predictions of the pretrained foundation module g(-) when it is fed with the refurbish module's outputs and the desired target values of the mobility- mobility-impaired impaired individual Yamp. In this strategy, the computed correction templates are completely ignored, and the model is trained solely to reproduce the final desired target. The loss function becomes
Liorget
Σ Xomp-Wamp E Extrnio aam
199h((ampi (h):: 07) -ammpl
where Vtrain is the training dataset consisting of desired outputs for mobility-impaired individuals. The foundation module g(-) is pretrained using the data from multiple able-bodied subjects, and its parameters O are kept fixed. Only the parameters O₁, of the refurbish module h(-) are varied using this loss function.
Hybrid. This is a combination of the correction-based and target-based training strategies. The loss function used to train the mapping model h is weighted combination of the error between the predicted correction Xcorr (XXmmp) for the -impiire individual input Xamp and its computed correction Xeorr and the error between foundation module predictions on mapping model outputs, Damp g(h(Xamm (m)))) and the desired output values of the mobility-impaie individual Damp. Thus, the hybrid loss function is,
Liotal = Q Linput + 3 Learget
(10)
where a and 3 are the factors that influence the effect of input- loss and target-based loss respectively on the final loss. Based on which correction strategy is used, the hybrid strategy can take two forms hybrid (neighbor) and hybrid (inversion).
5
Experiments
5.1 Baselines
We assess our proposed method's effectiveness against three baseline approaches:
Cross-mapping (zero-shot transfer). Here, we directly apply the foundation model g(.) to predict motion variables for mobility-inpaired individual subjects, without retraining the foundation module or utilizing the refurbish module transform mobility-impaired individual inputs into clean data. This is akin to zero-shot transfer, where the pretrained model is used to predict output trajectories
7

---

00-
0.55
3.5
0.00
0.4
11
20.23
12.00.23
Figure 3: Effect of foundation model architecture on the performance for different training strategies. (task-shared: model with shared backbone and common prediction head for all tasks, task-specifie: model with shared backbone and task-specific heads)
1.0
Ineer enpomentia
relghtorr based
2 0.4
Ieplerd (nwighter))
(a=1)
600
0.0
0.1 0.2 0.3
train_size size
Figure 4: (Left) Effect of weightage 33 of the target-based loss on the performance of models with tase-fpec prediction heads trained with hybrid strategy for refurbishing The weightage a of input- based loss is set to be 1. The blue curve the left (a 1, 0) represents purely neighbor-based refurbishing. (Right) Effect of sequence length 2m - 1 and number nearest neighbors n on the performance of models with shared and task- prediction heads trained using input- input-based based and hybrid strategies. For hybrid strategy, a = and 3 = 20 was selected. The combination which gave the best prediction performance in each case is marked with red square (Please note that there exists multiple combinations which gave similar accuracies).
from impaired inputs without any training. Since no training was involved, it was not possible to report scores for cross-mapping with different training ratios.
Direct-mapping, In direct mapping, we used a refurbish module h(.) in front of the pretrained model g()). The model learns to directly map the impaired individual's 's inputs Xamp the desired motion variables Tamp- During learning, the pretrained model is frozen, whereas the refurbish module is tunable.
Fine-tuning (transfer learning). In finetuning, no refurbish module is used, and the foundation module g(.) pre trained able-bodied data is finetuned learn mapping from impaired individual inputs Xamp to the desired motion variables Damp.
5.2 Ablation results
Foundation module architectures. Our initial analysis focused how the architecture of the foundation module impacts prediction performance across various training strategies. As depicted Fig. 3 we examined the foundation modules equipped with task-specific and task-shared predic- tion heads in predicting desired motion variables for mobility-impaire individuals across different tasks. For direct mapping and refurbished inputs with target-based and hybrid correction, models featuring task-specific heads outperformed those with shared prediction heads when the training data was limited. However, as the training dataset size increased, the performance of both models converged to similar levels. Interestingly. a reverse trend emerged when mobility -imptte--- individual inputs were refurbished with target-based trained refurbish module. Models with shared prediction heads performed better for the smallest training data sizes, while those with task-specific layers per- formed better with larger training datasets. In summary, task-specific foundation models consistently demonstrated superior prediction performance when compared to their task-shared counterparts.

---

1.0
hyarid (naighhor)
hybrid (Invorsion)
target
0.9
50-
0.8
Baiining_tyye type
390
392
150
150
150-
direct mapping
0.7
hybrid (neighbor)
233
300
200
300-
200
hybrid (inversion)
0.6
230
230
232
300
300
300
300
300-
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
350
100
300
Figure 5: (Left) Performance of models trained with different strategies. For hybrid strategies, and 3 20 was selected. For neighbor-bast strategies, m and n were selected based on the best- best-performing values computed in the previous section. (Right) The correction template Xcorr computed using different strategies and the corresponding predictions from the refurbish module. For target-based strategy, no correction template was computed, and the visualization shows the output of the refurbish module in this case.
Effect of weightage 3 of target-based loss In the hybrid training strategy of the refurbish module, we adopted loss function that is a weighted sum of the correction- based and surget-based loss. We next analyzed how the weights assigned to the loss functions affect the performance of the task-specific foundation modules. We set the weightage for correction- based loss to be and varied the weightage/ /3 of the target-based loss. We found that the model performance increases for higher weightage of target-based loss for refurbish model training. However, as the weightage /3 increases, the prediction performance shows converging trend (Fig. leett..
Effect of sequence length 2m + 1 and number of neighbors n. In the nearest neighbor search method for computing the correction template X we proposed a strategy for using a sequence of desired motion variables for the mobility-impairee individual (yamp ... Vamps..... ... Vamp j compute the correction template x for the k-th -ompiie individual input Xamp. Another strategy that we propose is to use a e-neighborhood around the able-bodied input sample Xú that is closest to the input required to produce the desired mobility-impaire individual motion variable vimp at time point k. The correction template should be computed as a weighted sum of n nearest neighbors of X², within the e-neighborhood (where weights decrease linearly or exponentially as the distance from X&, increases). We investigated the effect of the sequence length 2m + 1 and number of nearest neighbors n used for computing the correction template on the performance of models in predicting the desired motion variables for mobility-impaired individuals across locomotion tasks (Fig. 4 right).. Most of the cases evaluated worked better than the basic case of single sample matching and single neighbor (2m + 1 = 1, = 1). For models with task-shared prediction heads, better performances were obtained with lower sequence length 2m + 1 and higher number of neighbors for both correction-base and hybrid methods. For models with task-specific prediction heads, correction- based correction performed better with larger sequence length and larger number of neighbors lead to better performance for whereas hybrid strategy worked better with smaller sequence length and larger number of neighbors. Interestingly, hybrid (neighbor) strategy did not require matching a larger sequence of outputs, possibly because it also makes use of target-based correction. Nevertheless, sampling multiple neighbors from the e-neighborhood of X ab was still necessary to improve the prediction performance. A exponentially decreasing weighting performed better for neighbor-based correction strategy, whereas a linearly decreasing weighting worked better for hybrid strategy. For further analyses and comparisons, we use the sequence length, number of neighbors' and weighting values that gave the best prediction scores with neighbor-bart and hybrid (neighbor) computed correction templates.
Training strategies. Finally, we examined the impact of various training strategies on model performance (Fig. leef and Tab. []. Across all these strategies, we observed a general improvement in prediction scores as the training sample size increased. There was a slight drop in accuracy when using very large training sample sizes, likely due to overfitting. Both the hybrid strategies hybrid (neighbor) and hybrid (inversion) - which combined correction- based and barget-based mapping, consistently delivered accurate predictions across different training sample sizes. Notably, for the

---

Table I: Coefficient of determination (It2) obtained with different training strategies for train sample ratio of 0.1
cross
direct mapping
fine tuning
neighhor scarch
network inversion
target based
hybrid (neighbor)
hyhrid (iaversion)
mapping
tuskshaaedd task-specific
-0.35 + 0.16
0.77 0.2
0.73 0.22 0.74 0.22
0.33 0.11
0.23 0.07 0.39 = 0.25
0.39 = 0.25
0.84 + 0.07
0.83 + 0.09 0.87 0.09
-0.32 # 0.48
0.71 = 0.14
0.19 + 0.22
0.61 + 0.22
0.86 + 0.07
smallest amount of training data tested, the hybrid strategy outperformed the direct mapping and fine-tuning approaches. With task- prediction heads, hybrid reprogramming showed slight improvement (~ 7%) over direct mapping, while for task-specific prediction heads, it exhibited a more pronounced improvement (~ 16%). These results suggest that our proposed input refurbishing strategy offers an efficient means to adapt a pretrained model for new scenarios, especially when dealing with limited training data.
While direct mapping performs well in predicting motion variables for the specific individual it was trained on, it requires larger dataset to achieve the same performance level as the model reprogramming approach. Additionally, direct mapping relies on limited data from individuals, reducing exposure diverse motion conditions and leading lim- ited generalization capabilities. This limitation is critical for mobility-impairee individual motion prediction models, as gait patterns may change as mobility-impaires individuals adapt to generated motion. This adaptation could result in improved compensatory motions and gait normalization. In contrast, an able-bodied model trained on a wide array of motion scenarios from various individuals may be more adaptable to evolving mobility-impaired individual gait patterns while accommodating the predicted joint motion required for prosthetic walking. Our approach also outperformed model fine-tuning in the low-data regime, showing an improvement of approximately ~ 11% for task-shared models and 13% for task-specific models, underlining the effectiveness of the proposed approach over other model repurposing methods like transfer learning.
Visualization of the computed correction templates Xeorr and corresponding predictions from the refurbish module (Fig. right) shows that, despite the refurbish module being a simple three-layer MLP (chosen for its lightweight and data-efficient properties), can reconstruct the correction templates with considerable accuracy. Interestingly, the correction templates computed by the neighbor search and network inversion strategies were notably different, despite having the same desired output. This discrepancy arises because neighbor search finds the closest match from available data points, introducing variability based on the dataset, while network inversion uses gradient-based optimization to iteratively minimize error, leading to different solutions. The same variability was observed with target-based mapping, which directly optimizes towards the target output without generating intermediate correction templates.
Limitations The desired motion variables for mobility-impairee individual subjects were derived from able-bodied individuals with similar anthropometric features and walking speed as that of the mubility-impairee individual subjects. While this approximation may effectively represent the reference joint's desired motion, its suitability for practical applications such as controlling powered prostheses requires validation. Nevertheless, our approach possesses a generic capability to transform inputs in a way that can generate any desired output from a pretrained model, provided there exists a learnable relationship between the inputs and the desired outputs.
Conclusion and Broader Impact
In this study, we proposed ReMAP, model repurposing strategy that leverages deep learning's 's repro- gramming property, incorporating network inversion principles and retrieval-augmented mapping. Our approach adapts models originally designed for able-bodied individuals to forecast joint motion in limb-impaired patients without altering model parameters. Our findings indicate that the proposed input refurbishing strategy offers sample-efficien mechanism for adapting pretrained models to new scenarios. The proposed model reprogramming approach for adaptive motion forecasting has the potential to significantly enhance the quality of life for individuals with mobility impairments. By leveraging well-trained models from able-bodied data, this method can efficiently predict joint motions for mobility-impaire individuals, aiding in the development of more responsive and accurate
10

---

assistive devices such as prostheses and orthoses. This approach minimizes the need for extensive retraining, making it both cost-effective and accessible for personalized healtheare applications. Furthermore, this technique is versatile and can be applied to other regression problems in domains where data scarcity presents a challenge.
Acknowledgements
The authors express their gratitude to Prof. Dr. Arndt F. Schilling (University Medical Center Göttingen, Georg-August University Göttingen). Dr. Jennifer Ernst (University Medical Center Göttingen, Georg-Auguss University of Göttingen), Dr. Massimo Sartori (University of Twente), Dr. Thomas Schmalz, Dr. Michael Ernst (Department of Clinical Research & Services, Ottobock SE & Co. KGaA, Germany), Dr. Takashi Yoshida (University Medical Center Göttingen, Georg- August University of Göttingen), and Dr.Mahdy Eslamy (Teesside University), for guidance, helping with experimental data collection involving lower-limb impaired subjects, and the writing of the ethics. The lower-limb impaired individuals' data collection was carried out under grant number INOPRO-16SV7657 supported by the Federal Ministry of Education and Research (BMBF) https de/ hhee/rojjeke//nooggee
www..interaktiie-teechooo
References
[1] Joerg Kindermann and Alexander Linden. Inversion of neural networks by gradient descent. Parallel computing, 14(3):277-286, 1990.
[2] Navid Ansari, Hans-Peter Seidel, Nima Vahidi Ferdowsi, and Vahid Babaei. Autoinverse: Uncertainty aware inversion of neural networks. Advances in Neural Information Processing Systems, 35:8675-8686, 2022.
[3] Pirzada Suhail, Supratik Chakraborty, and Amit Sethi. Network inversion of binarised neural nets. arXiv preprint arXiv:2402./199) 2024.
[4] Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey, ACM Transactions on Information Systems, 1-64) 2024.
[5] Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Xin Jiang, Qun Liu, and Lei Chen. Retrieval- based disentangled representation learning with natural language supervision. In The Twelfth International Conference on Learning Representations, 2023.
[6] George-Sebastial Pirtoacă, Traian Rebedea, and Stefan Ruseti. Improving retrieval-based question answering with deep inference models. In 2019 International Joint Conference Neural Networks (IJCNN), pages 1-8. IEEE, 2019.
[7] Kailash Hambarde and Hugo Proenca. Information retrieval: recent advances and beyond. IEEE Access, 2023.
[8] Gamaleldin F Elsayed, lan Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks. arXiv preprint arXiv: 1806. /46, 2018.
[9] Yun-Yun Tsai, Pin-Yu Yu Chen, and Tsung- Yi Ho. Transfer learning without knowing: Repro- gramming black-box machine learning models with scarce data and limited resources. In International Conference on Machine Learning, pages 9614-9624. PMLR, 2020.
[10] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.0011 2021.
[11) Chao-Han Huck Yang, Yun-Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. arXiv e-prints, pages arXiv-2106, 2021.
[12] Hao Yen, Pin-Jui Ku, Chao-Han Huck Yang, Hu Hu, Sabato Marco Siniscalchi, Pin- Chen, and Yu Tsao. Neural model reprogramming with similarity based mapping for low-resource spo- ken command elassification. In Annual Conference of the International Speech Communication Association, 2023.
11

---

[13] Tuan Dinh, Daewon Seo, Zhixu Du, Liang Shang, and Kangwook Lee. Improved input reprogramming for gan conditioning. arXiv preprint arXiv:2201.02692 2022.
[14] Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of- --iittrrbtsttntiinnnnnn detection. Advances in Neural Information Processing Systems, 35: 1545551555557, 2022.
[15] Igor Melnyk, Vijil Chenthamarakshan, Pin- Yu Chen, Payel Das, Amit Dhurandhar, Inkit Padhi, and Devleena Das. Reprogranming pretrained language models for antibody sequence infilling. In International Conference on Machine Learning, pages 24398-24444 24419. PMLR, 2023.
[16] Lingwei Chen, Yujie Fan, and Yanfang Ye. Adversarial reprogramming of pretrained neural networks for fraud detection. Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2935-2933 2021.
[17] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 1312.6034, 2013.
[18) Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by invert- ing them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3188-5196, 2015.
[19] Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4829 4837,48777280 2016.
[20] Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint 2016.
[21] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imag translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[22] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.049966 2020.
[23] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmente generation for knowledge-nnteniiee nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[24] Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282. 2020.
[25] Erwin Aertbeliën and Joris De Schutter. Learning predictive model of human gait for the control of lower-limb exoskeleton. In 5th IEEE RAS/EMBS International Conference Biomedical Robotics and Biomechatrunics, pages 520-525. IEEE, 2014.
[26] Fabian Horst, Sebastian Lapuschkin, Wojciech Samek, Robert Müller, and Wolfgang I Schöllhom. Explaining the unique nature of individual gait patterns with deep learning. Scientific reports, (1):2339, 2019.
[27] Sharmita Dey, Sabri Boughorbel, and Arndt Schilling. Learning a shared model for motorized prosthetic joints to predict ankle-joint motion. arXiv preprint
2021.
[28] Erika V Coehler-McNichhee and Scott A Beardsley. Continuous myoelectric prediction of future ankle angle and moment across ambulation conditions and their transitions. Frontiers in Neuroscience, 15:709422, 2021.
Lara Riem, Philip Voglewede, Barbara Silver-Thorn, Sara R
12

---

[29] Zachary Choffin, Nathan Jeong, Michael Callihan, Edward Sazonov, and Seongcheol Jeong. Lower body joint angle prediction using machine learning and applied biomechanical inverse dynamics. Sensors, 23(1) 23(1):228, 2022.
[30] Sharmita Dey and Arndt F Schilling. A function approximator model for robust online foot angle trajectory prediction using a single imu sensor: Implication for controlling active prosthetie feet. IEEE Transactions Industrial Informatics, (962):447-14777 2022.
[31] Sharmita Dey, Mahdy Eslamy, Takashi Yoshida, Michael Ernst, Thomas Schmalz, and ArndtF Schilling. A support vector regression approach for continuous prediction of ankle angle and moment during walking: An implication for developing a control strategy for active ankle prostheses. In 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR), pages 727-733. IEEE, 2019.
[32] Feng- Yan Liang. Fei Gao, Junyi Cao, Sheung- Wai Law, and Wei-Hsip Liao. Gait synergy analysis and modeling on amputees and stroke patients for lower limb assistive devices. Sensors, 22(13):4814, 2022.
[33] Minjae Kim and Levi J Hargrove. A gait phase prediction model trained on benchmark datasets for evaluating controller for prosthetic legs. Frontiers in Neurorobotics, 16: 16:1064313, 2023.
[34] Sharmita Dey, Takashi Yoshida, and Arndt F Schilling. Feasibility of training random forest model with incomplete user-specific data for devising a control strategy for active biomimetic ankle. Frontiers in Bioengineering and Biotechnology, 8:855, 2020.
[35] Blair Hu, Elliott Rouse, and Levi Hargrove. Benchmark datasets for bilateral lower-limb neuromechanical signals from wearable sensors during unassisted locomotion in able-bodied individuals. Frontiers Robotics and AI, 5:14, 2018.
[36] Sharmita Dey, Takashi Yoshida, Robert H Foerster, Michael Ernst, Thomas Schmalz, Ro- drigo M Carnier, and Arndt F Schilling. A hybrid approach for dynamically training torque prediction model for devising a human-machine interface control strategy. arXiv preprint arXiv:2110.03085, 2021.
[37] Scott L Delp, Frank C Anderson. Allison S Arnll. Peter Loan, Ayman Habib, Chand T John, Eran Guendelman, and Darryl Thelen. Opensim: open-source software to create and analyze dynamic simulations of movement. IEEE transactions on biomedical engineering, 54(11):19400199) 2007.
[38] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint 1803.11803.0127 2018.
13

---

A Appendix
A.1 Network inversion
Starting with random input X ~ (/0 1), the gradient of loss, (X)(X), [amp)) of the foundation module output computed with respect to its input X. At each iteration t, the input is updated using gradient descent,
0c
x(++1= = = x + ax
(11)
where 7) is the learning rate and L is mean-squared error loss function, until convergence condition, (X),,, Jamp) < T, is satisfied. We used 17 = 0.01, r = le 4, and À 1.
A.2
Theoretical basis
In line with [ define the population risk for the target task muobility-impaire individual's motion prediction) via reprogramming a pretrained source network (able-bodied model) to be upper bounded by the sum of two terms:
1. Source population risk: The risk associated with the source task, which is denoted as (f).
2. Representation alignment loss: The Wasserstein-1 distance between the distributions of the source data (computed able-bodied template, X²) and the reprogrammed target data (h(X))).
Theorem Statement: Let h denote the learned additive input transformation for reprogramming and is the pretrained model. The population risk for the target task Re is upper bounded by:
(h) < R.(S) + W1(X,, (x*x,,(Xx')
(12)
However, the second term in the risk function is strict constraint and slight deviation in the representational alignment can lead a large error in the output. The wwworkk-iversiine based loss function relaxes the constraint such that the reprogrammed target data no longer needs to be similar to the source data. However, the constraint now changes to representational similarity between reprogrammed target data and network inversion inputs such that
(h) < Rf( + Wi(f (v'))
(13)
On the other hand, adding the target- based loss effectively relaxes this constraint by depending more on the source population risk and less on the stricter constraint of representational similarity of reprogrammed target data and source data. The target population risk becomes
(h(h) < (f. oWi(f (g')
(14)
with + 3 1 and B. Since the source population risk depends the performance of the pretrained model f, the population risk of the hybrid approach can effectively approach a lower error bound than 12 and B
A.3 Experimental details
Foundation module. The foundation module, g, consists shared core gs and lightweight task- specific prediction heads 9t- The shared core consists of two time convolution layers, with kernel size of 5 and dropout rate of 0.2. Each prediction head is a lightweight two-layer MLP with 200 and 100 units, with ReLU activation in hidden layers. The foundation module trained using able-bodied data from a publicly available dataset [35 with batch size of 100 for 25 epochs with stochastic gradient descent optimizer (learning rate of 1e-3 and momentum=0.9).
Refurbish module. Since we require the refurbish module be data-efficient, simple three-layer MLP with 100 units each with ReLU activation used. The model is trained for 100 epochs with
14

---

batch size of 100. A stochastic gradient descent optimizer with a learning rate of le-4 and momentum of 0.9 was used. Compute resources. All experiments were compatible on single GPU (Nvdia GTX3070, 8GB) with 12 CPU cores (Intel i7).
A.4 Extended results
Optimal Beta value
We evaluated the performance of models for different beta values and found that the best performance saturates at a 3 of 20 and diminishes for larger values 3.
obtained with 3 20 which was eventually selected. The performance
8
5
10
20
30
40
50
R²
0.63#0.04 0.88+0.02
0.92+0.02
0.94+0.01
0.94+0.02
0.90#0.05
0.85#0.07
Correction-basse mapping the below table, we report the correction based mapping results for different training ratios. It can be observed that for small amount of training data, the correction based method does not work well. However, as the amount of training data, the correction- performance also increases (as the refurbish module becomes increasingly accurate). These results show that the performance of the correction-bard based mapping depends strongly on the accuracy of the refurbish module. By adding the aageet-based loss, this constraint is relaxed and the model now depends more on the performance the pretrained model.
Train size
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Output
0.1840.14 20400006
0.4120.12
0,4820.11
0.5840.11
0.5740.11
0.57.0.16
0.6620.09
0.63a0.07
0.57,0.17
Refurbish model n!a
0.38+0.05
0.42=0.06
0.4640.05
0.45+0.05
0.48:0.06
0.48=0.09
0.48+0.05
0.45x0.09
A.5
User study
All the user studies presented here were approved by the Institutional Review Board (IRB) of the University Medical Center Göttingen. The participants gave their written consent to the study. Below you can find the snapshots of the general instructions given in the participant study, potential risks. precautions, and details about compensation.
15

---

General instructions' snippets of relevant information given to participants during locomotion experiments (both English- translated and German version below)
Dear Study Participant,
We hereby ask you to participate in our research study, carried out in the Applied Surgical and Rehabilitatio Technology Lab, Department of Trauma Surgery, Orthopaedics and Plastic Surgery. Your participation is voluntary and you can take back your consent at any time without giving any reason and without any influence on your current or future medical treatments.
Before submitting your consent, you have the right to be fully informed about the study.
Sehr geehrte Studienteilnehmerii,, sehr geehrter Studienteilnehmer,
hiermit bitten wir Sie um die Teilnahme einem unserer Forschungsprojekte, die im Applied Surgical and Rehabilitation Technologies Lab der Klinik für Unfallchirurgie, Orthopädie und Plastischen Chirurgie durchgeführt werden. Ihre Teilnahme ist freiwillig und Sie können Ihre Zustimmung jederzeit ohne Angabe von Gründen und ohne Einfluss auf derzeitige oder zukünftige Behandlungen widerrufen.
Zur Abgabe lhrer Einverständniserlläunng haben Sie das Recht Ihre Entscheidung zu bedenken, sowie eine Vertrauensperson während der mündlichen Aufklärung hinzuzuziehen.

---

Goal of the study
After a leg amputation, the amputated leg can be replaced by a prosthesis. But, we do not know what leads to the best use of prosthesis. The use of prosthesis can be influenced by surgical techniques, functions of the prosthesis, and the type of rehabilitation. this study, we aim to find out what happens the way people move after leg amputation. For this purpose, we will examine how people with an amputated leg perform tasks, which are related to walking, standing, and sitting. For comparison, will examine people that have not experienced amputation during the same tasks. In this study, we also aim to find out what happens to a leg after an amputation. For this purpose, we will examine the amputated leg in detail. Lastly, to better understand your situation, we will ask you to complete questionnaires about your amputation (if applicable), quality of life, and how physically active you are. All of the methods in this study are non-invasive.
segments, and range of motion. We will also attach electrodes and markers to your body for Parts 3 and 4. Parts and 4 (Movement Examination and 2), we will ask you to perform tasks that involve sitting, standing, and walking. You have the right to refuse any tasks.
Part 5 (Physical Examination 2) will be performed on a different day from Parts 2, 3, and 4. Part 5 will only be performed for people with amputated legs. In Part 5, we want examine whether people can feel gentle stimulation of the amputated and non- amputated legs. The gentle stimulation will include light touch, gentle vibrations, light pressure, and low-intensity electrical stimulation. All forms of stimulation are non- invasive.

---

Part 1: Interview
Part 2:
Part
Part 4: Movement Examination 2
Part 5: Physical Examination 2
Physical
Movement Examination Standing
Examination 1
Questions about your health, medications, amputation, prosthesis
Height
Walking on treadmill
Palpation of the amputated leg
Weight
Maximum contraction of muscles
Walking flat surface
Examination of sensations and perfusion of the amputated leg
Length of body segments
a) Comfortable walking speed b) Faster walking
Questionnaires about your quality of life and mobility
Sitting down and standing up
Range of motion at the hip, knee, and ankle
speed
Measurements the amputated leg
c) Slower walking speed
Walking on ramps
Ultrasound of the amputated leg
Photographs
Walking on stairs
Attach markers and electrodes for Parts and 4 Minutes
Photographs
Minutes
Minutes
155 Minutes
145 Minutes
Regular breaks are planned and breaks on demand can be inserted at any time.
Ziel der Studie
Eine Prothese ersetzt nach einer Amputation verlorene Funktionen des verlorenen Teils des Beins, wie zum Beispiel die Fähigkeit Gehen. Welche weiteren Auswirkungen durch die Amputation auf benachbarte
Gelenke und den Amputationsstumpf auftreten oder welche Prothesenfunktionen oder Rehabliitationsstattgiien zu einer optimalen Benutzung der Prothese führen unklar. Daher ist das Ziel dieser Studie die Beurteilung typischer Veränderungen, die nach einer Amputation im Gangbild und am Stumpf auftreten. Hierfür werden Menschen, die amputiert sind im Ganglabor beim Gehen untersucht. Zudem wollen wir Ihren Amputationsstumpf und das gesunde Bein untersuchen und vergleichen. Untersucht werden zum Beispiel Empfindungen am Stumpf (z.B. kalt, warm, Schmerzen, Missemppindungen) oder Versorgung der Haut des Stumpfes mit Sauerstoff. Zum besseren Verständnis lhrer Situation werden wir Ihnen Fragen stellen und Fragebögen austeilen.

---

Die Studie ist in 5 Teile aufgeteilt. Die Dauer der Einzelteile der Studie konnen Sie aus der Tabelle entnehmen. Am Anfang jedes Untersuchungsteils gibt es eine kurze Einführung in das Protokoll und mögliche Risiken der Prozedur. Zu diesem Zeitpunkt sowie während des ganzen Versuchs haben die Probanden die Möglichkeit, Fragen zu stellen und das Experiment abzubrechen.
Bei den beschriebenen Methoden handelt es sich um medizinische Standardverfahren, wie sie im täglichen Klinikbetrieb verwendet werden. Im ersten Teil (Interview) werden Sie befragt und gebeten Fragebögen zu vervollständigen, die wir Ihnen vorab per Post zugestellt haben. Die Teile 2, und 4 werden am selben Tag durchgeführt. Dann werden einige Untersuchungen (körperliche Untersuchung 1) zu Ihrer Körperstatik an Ihnen im Liegen und Stehen durchgeführt.
Im dritten und vierten Teil des Versuchs (aktive Untersuchung 1&2) werden Winkelsensoren und reflektierende Marker an Ihnen befestigt, durch die wir mit
Kameras Ihr Bewegungen aufzeichnen und genau nachvollziehen können wie sie mit Ihrer Prothese gehen.
Teil wird an einem anderen Tag als Teil 2, 3 und 4 durchgeführt. Teil wird nur für Menschen mit amputierten Beinen durchgeführt. Im letzten, dem fünften Teil des Versuchs (körperliche Untersuchung 2) möchten wir untersuchen, ob an bestimmten Stellen der Hautoberfläche des Stumpfs Empfindungen für das amputierte Gliedmaß ausgelöst werden können. Dies wird durch leichte Berührung mit Wattestäbchen systematisch untersucht. In einem zweiten Schritt werden zwei Arten taktiler Stimulation benutzt, um den Tastsinn anzuregen. Einerseits werden durch kleine Elektromotoren sanfte Vibrationen und/oder ein leichter Druck auf der Haut erzeugt. Diese Elektromotoren werden mit einem elastischen Band am Stumpf und dem gesunden Bein angebracht. Dieser Vorgang ist vollkommen schmerzfrei und ohne bleibende Schäden. Als andere Stimulationsart wird eine elektrische Stimulation mit Hilfe kleiner Elektroden erzeugt. Die selbstklebenden Elektroden werden auf der Hautoberfläche angebracht. Diese Art der Stimulation wird als ,elektrokutane Stimulation" bezeichnet, weil ein Strom geringer Intensität oberflächlich die Haut stimuliert. Die ausgelösten Empfindungen können einer Kombination des Klopfens, Drucks, und/oder des Vibrierens entsprechen. Bis zu 16 kleine Motoren werden an Gliedmaß- und Stumpfarealen angebracht. Die beiden erwähnten Methoden, d.h. elektrische und mechanische Stimulation, beruhen auf allgemein verwendeten Methoden, um gefühlsääßige Rückkooplunggsignalee im geschlossenen Regelkreis zu

---

liefern. Diese Methoden wurden bereits in mehreren vergangenen Studien sichel angewendet. In dieser Studie möchten wir testen, wie empfindlich die Probanden an den stimulierten Stellen sind. Genauer geht es darum unterschiedlich starke Aktivierungen einzuordnen (kaum spürbar, klar wahrnehmbare und fast unangenehme Vibrationen).
Wir würden uns sehr freuen, wenn Sie die Fragen und Fragebögen währenc unseres Versuchs zu Ihrer Person und Ihrer körperlichen Verfassung beantworter
würden. Wenn Ihnen eine Frage unangenehm ist, steht es Ihnen selbstverständlich frei, darauf nicht zu antworten.
(1) Interview
(2) körperliche Untersuchung 1
(3) Aktive Untersuchung
(4) Aktive
(5) körperliche Untersuchung
Untersuchung 2
2 Abtasten des Stumpfs
Fragen zu lhrer Gesundheit, Medikamenten, die Sie nehmen, Amputation und Ihrer Prothese
Größe
Ruhiges Stehen
Gehen auf einem Laufband
Gewicht
Maximale Muskelkraft Fuß und Bein
Körperstatik und Haltung Messen der Körperteillängen
Gehen auf einer ebenen Fläche
Untersuchung
von Empfindungen und Durchblutung am Stumpf
a) selbstgewählte Gehgeschwindigkeit
Hinsetzen & Aufstehen
Fragebögen zu Ihrer Lebensqualität und Mobilität
b) Schnellere
Bewegungs- freiheit der Hüft- Knie- und Sprung-gelenke
Gehgeschwindigkeit
c) Langsamere
Gehgeschwindigkeei
Vermessen des Stumpfs
Gehen auf einer Rampe
Fotodoku- mentation
Ultraschall des Stumpfs, Fotodoku- mentation Stumpf (2)
Treppen gehen
Anbringen der Marker für Teil 3,4 90 Minuten
45 Minuten Regelmäßige Pausen sind geplant und Pausen bei Bedarf können jederzeit eingefügt werden
30 Minuten
155 Minuten
145 Minuten

---

Risiken
Die Risiken der zuvor beschriebenen Schritte sind minimal.
Die nicht-invasiven Schritte zur Applizierung der Oberflächenelektroen (Stimulation und EMG) können zu leichtem Unbehagen durch Abrieb an der Haut führen. Das zur Verbesserung des
Haut-Elektroden-Konnttttss verwendete Hautreinigungsgel kann in sehr seltenen Fällen zu allergischen Reaktionen in Form von lokalen Rötungen führen. Manche Probanden können die Stimulation mit leichtem Strom/Vibration/Druck als unangenehm empfinden. Speziell für den Fall der Elektrostiiulattion kann die Haut unter der Stimulationselektrode eine leichte Rötung aufweisen. Nach Unterbrechung der Stimulation ist die Rötung im Normalfall nach 30- 60 Minuten nicht mehr vorhanden. Selten kann eine allergische Reaktion durch die Elektrostiiulation entstehen (lokale Schwellung oder Rötung). Wenn die
Elektrostiiulatiinn nicht ordnungsgemäß durchgeführt wird, kann diese als
Versuchsaufbau verwendet Standardverfahren mit minimaler Stimulationsintensstttß, wodurch dieses Risiko minimiert wird.
schmerzhaft empfunden werden.
Unser

---

Personal data
During the study, all of the data that we collect from you will be pseudonymized. This means that your data will be labeled with a randomly generated code, not with your name. Only the study investigators will know which code applies to your data. Also, your face will not appear in any of the photographs that we take.
For this study, we are partnering with Otto Bock GmbH (Germany) and the University of Twente (The Netherlands). Our partners will only receive anonymized data. This means that our partners cannot trace the data to your identity. For research
purposes, we may send the collected data to third parties (people or institutions that are not the study investigators). Third parties will only receive anonymized data. For publishing our findings in academic journals, we will only use anonymized data. you decide to take back your consent or withdraw from the study, your data will be deleted. After this study is terminated, the pseudonymized data will be stored for 10 years.

---

Persönliche Daten
Während der Studie werden personenbezogene Gesundheitsdaten erhoben und in pseudonymisierter Form aufgezeichnet. Pseudonymisierung bedeutet Verschlüsselung von Daten ohne Namensnennung nur mit Nummern codiert. Die Zuordnung der Daten oder Proben zu einer Person ist nur möglich, wenn hierfür der Schlüssel eingesetzt wird, mit dem die Daten pseudonymisiert wurden. Die personenbezogenen Daten/Proben werden unter besonderen Schutzvorkehrungen getrennt von der Schlüsselliste aufbewahrt. Eine Entschlüsselung nur durch die für die Studie verantwortlichen Personen möglich. Dritte erhalten keinen Einblick in die Originalunterlagen. Die Datenspelcherungszelt beträgt 10 Jahre. Eine Datenweitergabe pseudonymisierter, das heißt verschlüsselter, Daten ist mit den Kooperationspartnern der Firma Otto Bock und der Universität Twente in den Niederlanden geplant. Dies betrifft Daten aller Teilexperimente. Dabei ist sichergestellt, dass der Forscunngsparee die Daten nicht auf einzelne Personen rückverfolgen kann. Bei Widerruf der Studienteilnahme werden die personenbezogenen Daten gelöscht. Eine Veröffentlichung findet nur mit anonymen Daten statt. Die experimentellen Ergebnisse können einer dritten Person für Forschungszwecke weiter gegeben werden. In diesem Fall werden die Daten verschlüsselt und der Privatbereich der Teilnehmer völlig geschützt sein.
Participating in the Study
To participate in this study, you must be willing and legally able to provide your written informed consent. Your eligibility will be checked by medical doctor. If any of the following conditions applies to you, you cannot participate in the study:

---

Medical conditions that cause visibly abnormal movement
Examples: Broken bones, pain
A major amputation other than an above-knee, below-knee, or through-knee amputation on one side
Dementia
Open wounds, swelling, infection, or pain in the amputated leg
Risk of stress fractures or fractures from low-impact falls
Pregnancy
Addiction to drugs or alcohol
Inability to communicate with the investigators
Lack of cooperation with the investigators
Medical conditions that make participation dangerous
Example: cardiovascular disease
Teilnahme an der Studie
Voraussetzung für die Teilnahme an der Studie ist die Einwilligung und Einwilliggunssähiikeit.. Die Ein- und Ausschlusskriterien werden von einem Arzt
überprüft. Sollte eine der folgenden Punkte auf Sie zutreffen, ist eine Teilnahme an der Studie nicht möglich:
Premature Interruption of the Study
If you experience any discomfort during the study, you have the right to stop participating in the study. If you develop any condition that makes you ineligible for the study while you are participating, the investigators will stop your participation. You have the right to stop your participation at any time for any reason. your participation stopped prematurely, you will be paid for the time you have spent.
Approval of the Study
All procedures for this study have been approved by the responsible ethics committee under the processing number XXXXX..

---

Reimbursement
You will be reimbursed at E10 per hour for your participation. This includes the time required for transportation to and from the study.
Further Questions
If you have any further questions, please do not hesitate to contact us.
Your Study Team

---

Abbruch der Studie
Falls bei Ihnen unerwartet Rötungen, allergische Reaktionen, Herz- Kreislauf Symptome während des Versuches auftreten oder Sie im Verlauf Ausschlusskriteriee aufweisen, dann kann jeder einzelne Versuchsschritt jederzeit abgebrochen werden. Sie erhalten dann eine Bezahlung entsprechend dem Zeitaufwand, dem Sie tatsächlich teilgenommen haben. Sie können zu jedem Zeitpunkt die Teilnahme am Forschungsprojekt abrechen. Natürlich würden wir dann gerne von Ihnen wissen wollen, warum Sie die Teilnahme abbrechen. Aber Sie brauchen uns keine Gründe nennen, wenn Sie nicht möchten.
Genehmiqung des Versuchs
Die Versuchsdurchtühnung die wir lhnen hier vorstellen, wurde von der zuständingen Ethik-Kommission unter der Bearbeitungsnummer 26/3/18 zustimmend bewertet.
Autwandsentschädlguun
Zur Aufwandseetschádiguug für die Teilnahme an der Studie erhalten Sie einen Betrag von E/h. Im Falle eines vorzeitigen Abbruchs der Studie erhalten Sie ebenfalls eine Aufwandsentschääigung, entsprechend der bis dahin beanspruchten Zeit. Die Anreisezeit wird ebenfalls mit 10 E/h entschädigt.
Weitere Fragen
Sollten Sie weitere Fragen haben, zögemn Sie nicht die Prüfdurchführenden anzusprechen.
Bei Fragen stehen wir Ihnen gerne zur Verfügung.
lhr Studien-Team

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose data-efficient model re-purposing approach by which a model trained for motion prediction in able-bodied subjects could be adapted for individuals with limb loss. Through extensive comparisons, we show that our proposed approach outperforms other baseline and state-of- ofce-oo-lllo the-art approaches, shown in Fig. 3 and Tab.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have included Limitations section in the manuscript which discusses the limitations of our study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We have detailed the approach in the Methods section with details on the network used and the hyperparameters for the network and the proposed retrieval-based template matching strategy.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The data is related to consortium project and requires permission from all involved parties.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppp.cc/pubcbc/uuuceeee Codeeubmii ssiinn criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the Methods and Appendix sections of the paper, we have provided the details regarding the training.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Although we did not perform statistical significance analysis, we have provided error bars in the results wherever it is applicable. In one of the figures (Fig. A right), we could not provide error bars due the two-dimensional structure of this figure.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details the compute resources provided in the appendix. We did compute the time requirements during our experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have carefully read the NeurIPS Code of Ethics and reviewed that work conforms with it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have included section on Broader Impact which discusses the positive societal impact of our study. We do expect any negative societal impacts of study.
Guidelines:
30

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have used publicly available dataset of able-bodied locomotion [35] (License: CC BY 4.0) to train our foundational module. We have explicitly mentioned this in the Datasets section and provided citation the related paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have provided the full text of instructions given to participants and screen- shots and the details about compensation in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We provided the snapshots of the participant information and potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "KR6UhajHoD", "reviewer": "Reviewer_rCTE", "summary": "The paper proposes a novel method, ReMAP, to address the challenge of motion prediction for individuals with limb loss, particularly in scenarios with limited data. The authors introduce a model reprogramming strategy that leverages deep learning's ability to adapt to new tasks without altering model parameters. The approach combines network inversion principles and retrieval-augmented mapping to repurpose models trained on able-bodied individuals for motion forecasting in limb-impaired patients. The paper presents promising results, demonstrating the effectiveness of ReMAP through empirical studies on data from below-knee amputees. The findings suggest significant improvements over traditional transfer learning and fine-tuning methods, highlighting the potential of ReMAP to advance assistive technology and improve mobility for patients with amputations, stroke, or aging.", "questions": "Mentioned above", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper's findings have significant implications for advancing assistive technology and improving mobility for individuals with limb loss. By demonstrating the potential of model reprogramming in this domain, the authors open up new avenues for research and development, potentially leading to more effective and personalized prosthetic solutions.\n2. The methodology presented, while novel, is primarily geared towards limb motion prediction and seems to have limited generalizability beyond rehabilitation and robotics. Therefore, given the paper's specific focus on rehabilitation and assistive technologies, the authors might consider submitting their work to a more specialized venue, such as the International Conference on Rehabilitation Robotics (ICORR) or a conference on prosthetics and orthotics. This could potentially provide a more targeted audience and foster collaborations with researchers directly working in this field.", "weaknesses": "1. Source of Ground Truth for Amputees: The paper mentions that the ground truth ankle motion trajectories for amputees are computed based on similar-speed gait cycles of able-bodied subjects with comparable anthropometric features. This raises a fundamental question: if the model relies on able-bodied data for ground truth, why not directly use the foundation model trained on able-bodied data for prediction? The paper does not provide a clear explanation for this.\n2. Lack of Architectural Details: The paper lacks a detailed description of the foundation model's architecture. It mentions using time convolutions and task-specific layers but does not specify the input and output dimensions, or the number of layers. This lack of transparency makes it difficult to assess the model's complexity and suitability for the task. Furthermore, the paper does not provide any justification or experimental evidence for the chosen architecture, leaving the reader to wonder if alternative architectures might be more effective.\n3. Choice of R2 as Evaluation Metric: The choice of R2 (coefficient of determination) as the primary evaluation metric, which is a common choice for regression tasks. However, R2 primarily measures the proportion of variance explained by the model, which may not directly translate to the functional performance of a prosthetic device. Alternative metrics like mean absolute error (MAE) or root mean squared error (RMSE) could provide a more intuitive understanding of the average prediction error in joint angles, which is crucial for prosthetic control. Additionally, task-specific metrics like gait symmetry or measures of metabolic cost could offer more clinically relevant insights into the model's performance.\n4. Optimal Beta Value: In Figure 3, the performance improvement with increasing beta appears to plateau. Did the authors investigate an optimal value for beta beyond which performance gains diminish or become negligible? If so, what was the optimal beta, and how was it determined? If not, what motivated the choice of beta values presented in Figure 3?\n5. Target-Based Loss Limitations: While Figure 3 focuses on the positive impact of increasing beta, what are the potential limitations or drawbacks of relying solely on the target-based loss function? Does it lead to overfitting or other issues that might explain the lower R2 values observed in Table 1? Are the experimental setups in Figure 3 and Table 1 identical? If not, what differences in the datasets, train-test splits, or other factors could explain the discrepancy in the results? Were the same amputee subjects used in both experiments?\n6. The authors mention that \"for the smallest amount of training data tested, the hybrid strategy outperformed the direct mapping and fine-tuning approaches.\" but from table 1, it seems the simple baseline of direct mapping has considerable confidence interval overlap with the proposed approaches. Can authors please clarify this observation?\n7. Approximation of Desired Motion: The paper acknowledges that the desired motion variables for amputee subjects are derived from able-bodied individuals with similar anthropometric features and walking speed. While this approximation may be reasonable, it may not fully capture the unique motion patterns and compensatory mechanisms specific to individual amputees. The suitability of this approximation for real-world applications, such as controlling powered prostheses, needs further investigation.\n8. Limited Generalizability: The study primarily focuses on below-knee amputees, and the generalizability of the ReMAP approach to other types of limb impairments or a wider range of motion conditions remains unclear. Further research is needed to validate the effectiveness of ReMAP in more diverse patient populations (abled bodied subjects are only 10) and scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our manuscript and provide insightful feedback and suggestions to improve it. Below, we address your comments point-by-point. Please let us know if you have further questions and we will be happy to address them. Due to the rebuttal limit of 6K characters, we moved some tables under the \"official comment\" tab following this rebuttal.\n\n**Using foundation models directly on amputees** Thank you for your comment. We would like to clarify this further. We address this point empirically in **Table 1** of our paper *(cross-mapping)*, where we show that directly using the foundation model performs poorly. This is because the inputs from the residual body parts of amputees differ significantly from those of able-bodied subjects due to compensatory movements and other physiological factors. Consequently, using amputee measurements directly as inputs to the foundation model produces output motion trajectories that are significantly different from able-bodied references.\n\n\n**Architectural details:** We specified the architectural details of the foundation module and refurbish module in the appendix section. Given your comment, we are happy to move this to the main section and include details such as input/output dimensions. Our foundation module consists of a shared core that learns generic motion patterns, and task-specific heads that focus on the specifics of different locomotion tasks. This approach, similar to architectures like the Dinov2 model from Meta, demonstrates that combining a shared core with task-specific adaptations improves prediction accuracy compared to a shared model without task-specific layers (Fig. 2). For a detailed description of inputs and outputs of the model, we would like to refer to our answer to Reviewer 2Em3 under \"Description of Inputs/Outputs\". \n\nAdditionally, we present an evaluation of the performance of various time series forecasting architectures, including LSTM, Transformers, and TCN. Our findings indicate that TCN performs better overall in the context of this study. This result informed our decision to use a TCN-based architecture as the pretrained model. \n|              |      Accuracy      |\n|:------------:|:------------------:|\n|     LSTM     |      0.92±0.03     |\n|  Transformer |      0.93±0.02     |\n| **TCN** | **0.94±0.02** |\n\n**Target-based loss limitations:** Empirically, we found that using only the target-based loss function requires more training data to achieve the same accuracy as the hybrid loss function. Sole reliance on the target-based loss can lead to overfitting and reduced accuracy with small datasets. Additionally, models trained on small datasets with target-based loss may learn dataset noise, resulting in high variance, as observed in our results with a 0.1 training ratio for target-based loss. The $R^2$ values in Fig. 3 are averaged across all sequence lengths $m$ and delays $n$, while those in Tab. 1 are for the best $m$ and $n$ values. We will clarify this in the figure caption.\n\n**Accuracy of direct mapping and hybrid strategy:** As mentioned in our previous answer, the high variance of direct mapping (target-based loss) can be attributed to the model overfitting the noise in the small dataset. On the other hand, the hybrid strategy uses a combination of target-based and correction-based loss. Here, the correction-based loss effectively acts as a regularization, reducing the overfitting and variance. \n\n**Generalizability beyond rehabilitation and robotics:** In this study, our focus is on proposing a machine learning approach for repurposing a pre-trained model solely through input-level manipulations, particularly benefiting low-data regimes and multiple tasks. Thus, our method is applicable to different types of limb impairments, and is broadly applicable beyond rehabilitation and robotics, extending to data-scarce domains such as human-machine interactions, health applications, and problems requiring rapid adaptation to new environments, including autonomous driving, virtual reality, and industrial robotics. In our final manuscript, we will elaborate on this wider applicability, encouraging readers to consider broader uses. Further, the dataset that we used consists of around 650k samples per subject across different locomotion tasks – walking on level ground, stair ascent, stair descent, and different inclines – that are most commonly encountered during daily locomotion totaling 6.56M samples across ten subjects. Therefore, the current dataset contains good enough variations to validate our approach.\n\n**Approximation of desired motion:** Our decision to use the motion of able-bodied subjects to derive reference joint motions for amputees is inspired by previous studies in prosthetic control [1], which have also utilized able-bodied subjects' motions as approximate references. These studies have demonstrated that this approach is effective for generating control commands for prosthetic devices for amputee patients. We do plan to conduct future real-time experiments with prosthetic devices with amputee patients, pending ethics approval.\n\n[1] Gehlhar R, et. al. A Review of Current State-of-the-Art Control Methods for Lower-Limb Powered Prostheses. Annu Rev Control. 2023;55:142-164. doi: 10.1016/j.arcontrol.2023.03.003.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qke7OEFa9y", "reviewer": "Reviewer_2Em3", "summary": "This paper introduces ReMAP, a novel approach for adapting motion prediction models originally trained on able-bodied individuals to predict joint motion in limb-impaired patients, particularly those with below-knee amputations. The key innovation is the use of neural model reprogramming, which allows the adaptation of models trained on able-bodied individuals to limb-impaired patients without altering their parameters.\n\nReMAP attempts consists of three main components:\n1. A foundation module: A diverse model trained on able-bodied data.\n2. A retrieval-augmented template mapping module: For the input of a limb-impaired person, finds most similar inputs of able-bodied persons.\n3. A refurbish module: Given both the identified correponding input pairs as well as the to-be-predicted motions, train a network to map limb-impaired inputs to \"corrected\"/able-bodied inputs.\n\nFor training the refurbish module, the authors attempt \nSo overall, the method leverages network inversion techniques and retrieval-based methods to map amputee inputs to able-bodied patterns.  This approach aims to address the challenge of limited data availability for impaired patients while capitalizing on the rich data available for able-bodied individuals.\n\nThe researchers evaluated ReMAP against zero-shot, from-scratch-training and finetuning baselines amd found ReMAP to clearly outperform them", "questions": "Is it correct that if you add the regularization term $\\lambda ||X||_2$ into the target-based mapping you would have the same overall loss as hybrid(inversion), just trained end-to-end instead of in two stages? If so, would be really interesting to have that as another variant and see how different the result are.\n\nCould the whole network inversion stuff also lead to completely unrealistic/\"adversarial\" examples?", "limitations": "The authors do discuss the main limitation that the targets for limb-impaired patients were derived from able-bodied individuals.  As mentioned in weaknesses might be helpful to mention that these are the targets  in the introduction already.\nAlso, there is no external results to compare to, which is maybe owed to the particular application, but always makes it much harder to estimate the quality of the proposed method.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* valuable task with clear motivation for the proposed method\n* implemented several meaningful variants and baselines \n* very good results for the proposed method", "weaknesses": "While the authors also mention it in their limitation, it is a definitely a substantial issue that no real ground truth exists. Since the to-be-predicted motion trajectories are extracted from able-bodied subjects in this study, it might favor the proposed method. Maybe it could at least be mentioned in introduction already that this is how the actual targets are being computed.\n\nAlso with regard to the writing, maybe in Inputs/Outputs, already X and y could be defined, I took me longer than necessary to understand what exactly is being predicted from what, again by maybe giving an example already in the introduction and also explaining that target values for impaired patients are actually deduced from able-bodied data might make it easier to more quickly understand the paper.\n\nAnother comparison, where you add the regularization term $\\lambda ||X||_2$ to the target-based mapping variant would be useful (also see questions).\n\n\"namely nearest neighbor search and gradient-based optimization\" -> is followed up with a subheading \"nearest neighbour search\", but no subheading \"gradient-based optimization\"... this is a bit confusing\n\nPersonally, would not need the explanation of individual steps of gradient descent in network inversion part.\n\nFigure 1 font is quite small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wholeheartedly thank you for the positive evaluation of our work and the insightful feedback on our manuscript. We are happy that you recognize the value and motivation behind our proposed method, as well as the meaningful implementation variants and baselines. Your positive remarks are encouraging. Below, we address your comments point-by-point.\n\n- **Include ground truth computation in introduction:** We agree with you and will indeed mention the computation of the ground truth in the introduction. \n\n- **Definitions of X and y:** Thank you for the comment. We are happy to further clarify the inputs and outputs of the model. The inputs are the history of the angular velocities of the thigh and shank segments of both the left and right limbs and the left and right knee angles. The outputs are the ankle joint profiles of the amputated side. Formally, \n\n\\\\[X_t=\\\\{\\theta ̇_{(t-K:t)}^{(thigh,l)},\\theta  ̇_{(t-K:t)}^{(thigh,r)},\\theta  ̇_{(t-K:t)}^{(shank,l)},\\theta _{(t-K:t)}^{(shank,r)},\\theta _{(t-K:t)}^{(knee,l)},\\theta _{(t-K:t)}^{(knee,r)} \\\\} \\in \\mathcal{R}^{(K×D)}\\\\]\n\nand\n\n\\\\[y_t=\\theta _t^{(angle,amp)} \\in \\mathcal{R}\\\\]\n\nWhere $K=20$ is the length of history and $D=6$ is the number of input features. The target $y_t$ is computed from able-bodied individuals with similar anthropometry and walking speeds. \n\\\\[y_k^{amp}=\\frac{(\\sum_{(s\\in S_{anthropometric})}\\sum_{(n\\in N_{speed})}y_k^{(s,n)} )}{(|S_{anthropometric} ||N_{speed} |)}\\\\]\nWhere $ S_{anthropometric }$ is the set of subjects with similar anthropometry as the amputee (height: ±5cm, weight: ±5kg) and $N_{speed}$ is the set of gait cycles where the able-bodied subjects in $S_{anthropometric}$ walked at similar speeds (±0.1m/s) as that of the amputee. The reference time point $k$ for computing a matching amputee output was based on the phase of the gait cycle. As the reviewer suggested, we can include the details into the revised manuscript which we omitted earlier due to space constraints. \n\n- **Adding a regularization term to target-based mapping comparison:**  We thank you for this excellent suggestion. Given that the correction-based mapping loss effectively acts as a form of regularization, it is indeed useful investigating whether a direct regularization term on the foundation module input would improve the accuracy when training end-to-end using a regularization. As you suggested, we added a regularization term to the target-based loss as follows:\n\n$$ L_{target}=\\\\sum_{(X^{amp}, y^{amp})} ||g(h(X^{amp}, \\\\Theta_h ), \\\\Theta_g^* ) - y^{amp} ||_2 + \\\\lambda ||h(X^{amp}, \\\\Theta_h )||_2 $$\n\nWith $\\\\lambda\\\\in [10^{-6},10^{-4},10^{-2},10^{-1},1,10]$. However, adding the regularization term did not substantially improve the accuracy when using the target-based loss function for any lambda value. \n\n| $\\lambda$ | $10^{-6}$ | $10^{-4}$ | $10^{-2}$ | $10^{-1}$ |    $1$    |    $10$   |\n|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|   $R^2$   | 0.69±0.07 | 0.69±0.08 | 0.69±0.07 | 0.58±0.06 | -0.46±0.4 | -1.02±0.8 |\n\nThis result indicates that the input regularization achieved by adding correction-based loss outperforms other types of regularization. Nevertheless, we wholeheartedly acknowledge the value of your proposal as an important baseline to be included to improve our manuscript. \n\n- **Adversarial examples from network inversion:** Thank you again for the excellent question. As you correctly pointed out, network inversion using gradient backpropagation can lead to unrealistic inputs (adversarial examples) since it does not impose any constraint during network inversion. However, this does not pose a problem since the refurbish module ensures that an alignment is learned between the target representation space and network inversion inputs.\n\n-**Other comments** \n   - **Names of subheadings:** Thank you for pointing this out. We will adjust the organization of the subheadings.\n   - **Explanation of gradient descent in network inversion:** We will move this to the appendix.\n   - **Figure 1 font size:** We will make the fonts more legible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8omPuDMPhv", "reviewer": "Reviewer_Du29", "summary": "### Summary\n\nThe author(s) introduce a neural model reprogramming based approach for motion signals based time series forecasting, which is using input alignments method to match target low-resource patient data to the pre-trained models with better generalization. \n\nThe proposed alignment method mainly characterize two approaches on (1) using distance based retrieval and (2) network inversion optimization to improve general performance on the model reprogramming.\n\nIn general, these two directions are essential to bring low-resource or unseen shallow data to pre-trained models due to the information alignments. Indeed, the authors are unfortunately missing highly related published works (i.e., on the similarity distance measurement [A] and network inversion [B]) on this two direction on model reprogramming. \n\nBesides, when there are theoretical foundation and simple measurement tool based on the 1-d Lipschitz properties on the time series reprogramming in [11], it would be good to have some connections (e.g., use the distance of Lipschitz properties upper bound) on justifying the performance differences between these two solutions. \n\nI think the direction would be still with general interests to the NeurIPS community. It is a borderline paper to me. \nThis paper is also with a few formation and grammar issues; the authors need to carefully revise.\n\n***\n\n### References\n\nA. Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Classification, Interspeech 2023\n\nB.  Improved Input Reprogramming for GAN Conditioning, ICML 2022 Workshop", "questions": "1. Could we have more details on the pre-trained model used in the work? The information in the appendix is not clear. \n- Since \"foundation model\" refers to larger or more generalized methods compared to simple MLPs\n\n2. Do the author(s) evaluate other accessible time series foundation model with the proposed methods?\ne.g., Time-LLM in ICLR 2024\n\n3. Are there any justification between the selections of RAG-alike mapping and network inversion?\n- since the performance of these two approaches are really similar.", "limitations": "This work involves in some human studies. The author(s) had provided some human interface they used in the evaluation. \nI think there is no need for ethics review needed based on the algorithm is mainly on the alignment side.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### Pros.\n\n1. Both hybrid results outperform the fine-tuning baseline as empirical strong findings.  \n- It is recommended to add some complexity or training cost analysis\n\n2. The attempt on bridging time series processing and model adaptation is new as applications.", "weaknesses": "### Cons.\n\n1. The paper format needs to be improve before a final version, where many format issues occur in the current version\n\n- Section Indexes are missing (e.g., **1.** Introduction, ...)\n\n- The grammar and formation at the first few pages are acceptable but it seems to be losing its track from Page 5 line `179`\n\nFor example, as\":\" and given by\":\"\n\n\n2. Technically, there are some distance metrics based neural model reprogramming on similarity mapping and the theoretical approaches are less covered in the current work, which a little bit degrades the depth of this work. \n\n3. The term of foundation model seems to be a shallow neural network, where foundation model are usually denoted as general purpose model model with large parameter scale and low generalization loss due to its over-parameterization.\n\n4. The evaluation section is relatively sparse. It is not clear on its relationship to the other well-known benchmark on time series forecasting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive and thoughtful review. We appreciate your recognition of the impact of our work, your acknowledgment of our work as interesting to the NeurIPS community,  and your detailed feedback on areas for improvement. Below, we address each of your review comments. Due to the rebuttal limit of 6K characters, we move an answer to the \"official comment\" tab following this rebuttal.\n\n**Theoretical relation with [11]:** Thanks for the constructive suggestion to enhance the value of our work. In principle, the theoretical foundation of the error bound provided in [11] also holds for our approach of neighbor-search-based mapping, where the refurbish module should map the target template (amputee input trajectory) to the source template (able-bodied motion trajectory). Here, the population risk for the target task (amputee motion prediction) via reprogramming a pre-trained source network (able-bodied model) is upper bounded by the sum of two terms:\n- **Source population risk**: The risk associated with the source task, which is denoted as $R_S (f)$. \n- **Representation alignment loss**: The Wasserstein-1 distance between the distributions of the source data (computed able-bodied template, $X^s$) and the reprogrammed target data ($h(X^t)$).\n\\textbf{Theorem Statement:} Let $h$ denote the learned additive input transformation for reprogramming and $f$ is the pretrained model. The population risk for the target task $R_t (h)$ is upper bounded by: \n$$ R_t (h)≤ R_S (f) + W_1 (X^s,h(X^t)) \\\\qquad (1) $$\nHowever, the second term in the risk function is a strict constraint and a slight deviation in the representational alignment can lead to a large error in the output. The network-inversion based loss function relaxes the constraint such that the reprogrammed target data no longer needs to be similar to the source data. However, the constraint now changes to a representational similarity between reprogrammed target data and network inversion inputs such that\n$$ R_t (h)≤ R_S (f) + W_1 (f^{-1} (y^s),h(X^t)) \\\\qquad (2) $$\nOn the other hand, adding the target-based loss effectively relaxes this constraint by depending more on the source population risk and less on the stricter constraint of representational similarity of reprogrammed target data and source data. The target population risk becomes\n$$R_t (h)≤ βR_S (f) + αW_1 (f^{-1} (y^s),h(X^t))$$ \nwith $α+β=1$ and $α<β$. Since the source population risk depends on the performance of the pretrained model f, the population risk of the hybrid approach can effectively approach a lower error bound than (1) and (2).\n\n**Formatting issues:** Thank you very much for pointing them out. We will thoroughly revise and correct them before the final version.\n\n**Evaluating our proposed approach on other time series foundation, namely Time-LLM from ICLR 2024:** We thank the reviewer for this suggestion and acknowledge that such an evaluation would be interesting. However, the Time-LLM is trained on natural language, therefore, the method would need to transform our input time series data into text prototype representations along with prompts, which can be then processed by the LLM to guide the model toward generating forecasts. We are already working on it, however these changes would require a bit more time, and we are happy to include the results with the concurrent Time-LLM in our final version.\n\n**Pretrained model, naming:** Our pretrained model is a foundation module since it acts as a generalized base that learns diverse motion patterns across multiple able-bodied subjects with different anthropometry, performing various motion tasks at different modes, speeds, and inclines. The size of the model is designed to suit the dataset size and computational budget. This module consists of a shared core that learns generic motion patterns, and task-specific heads that focus on the specifics of different locomotion tasks. The shared core, comprising temporal convolution layers, handles sequential information and learns generalities across diverse motion conditions with shared weights. This creates a generalized foundation, leveraging diversities among tasks. To address task-specific requirements, we add lightweight task-specific layers, characterized by two-layer MLPs with ReLU activations, on top of the shared core. This approach, similar to architectures like the Dinov2 model from Meta, demonstrates that combining a shared core with task-specific adaptations improves prediction accuracy compared to a shared model without task-specific layers (Fig. 2). Thus, our model uses MLPs only for lightweight task-specific layers, while the shared core, trained on diverse motion conditions from multiple subjects, serves as a generalized foundation for different gait patterns.\n\n**RAG-like mapping and network inversion:** The main difference between these methods is in memory and computational requirements. Neighbor search requires access to the corpus of able-bodied inputs used for training, leading to a memory requirement of $O(N.D)$ (where $N$ is the number of training samples and $D$ is the input dimension). This approach can be faster for moderate-sized datasets. Network Inversion eliminates the need to store the entire dataset by computing correction templates through gradient optimization, resulting in a memory requirement of $O(D)$. This method is preferable under tight memory constraints and offers slightly higher performance benefits but involves iterative optimization, which can be slower.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ulAiO8BPcL", "reviewer": "Reviewer_wD18", "summary": "This paper present a model reprogramming-based strategy, ReMAP, to repurpose a foundation model, tasked with forecasting joint motion, pretrained with able-bodied (source) subjects  for amputee (target) subjects. The proposed method incorporates network inversion and retrieval-augmented mapping to identify the closest matching source sample for each specific target sample. ReMAP holds the promise to adapt pretrained large-scale foundation model to downstream tasks effectively, without the need to retrain/finetune extensive parameters of the foundation model. The empirical results are conducted with self-collected below-knee amputated patients.", "questions": "plaese see Weakness section", "limitations": "The authors do have include a limitation section, but do not clearly indicate what actionable can be done in future works.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. this research aims to address a critical problem in assistive technology, holding important societal implications to improve the quality of life for individuals with mobility impairments.\n2. The reprogramming-based adaptation is promising, especially in the current era of foundation models, considering its computationally efficient property.\n3. The authors collect a dataset with practical values and claim that they have the plan to open source the data.", "weaknesses": "1. While the methodology is generally followable, certain descriptions are unclear. This includes but is not limited to:\n    - what does it actually mean by task-shared and task-specific? Of course I understand their meanings in the general machine learning context, but I am not sure how are they reflected on this particular able-bodied dataset?\n    - The difference between direct-mapping and fine-tuning, since the definition of “user-specific models” is unclear to me.\n    - I also do not see where is the result for baseline “cross-mapping”, except for Table 1 in which only the results for 0.1 train ratio are reported?\n2. The reported numerical results for neighbor search and network inversion (Table 1) do not appear to have significance differences. How does it demonstrate the superiority of network inversion over neighbor search?\n3. As the authors have reported the target-based results, why do not report correction-based results as well? I was wondering how these two objectives contribute to the prediction, respectively - is there any observable or interpretable insights?\n4. The authors also mention that the performance of reprogramming is better than fine-tuning. This is a bit counter-intuitive and conflicts with the results in other reprogramming contexts (e.g., visual reprogramming). Can you provide more details of the comparison regarding the number of trainable parameters in two approaches?\n5. The authors do not clearly explain how $\\tilde{y}_{amp}$ is calculated from able-bodied individuals. And the template searching strategy assumes measurements alignment of able-bodied and amputee subjects for the same motion. However, this might not be true due to discrepancy and subject variability. For instance, when there is no “perfect” match in the able-bodied subject in the pool, is such distance-based matching still meaningful?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the impact of our work. Your questions and constructive suggestions have helped us improve its clarity. Below, we provide detailed answers to your comments point-by-point. Due to the rebuttal limit of 6K characters, we move some answers under the \"official comment\" tab, following this rebuttal\n\n**Clarity of descriptions:**\n- **Task-specific vs. task-shared :** The able-bodied dataset consists of various locomotion tasks: walking on level ground, slope ascent, slope descent, stair ascent, and stair descent from different subjects across different speeds. The task-shared model variant is a fully weight-shared joint model, where the model learns diverse locomotion tasks and motion conditions with parameters fully shared between the tasks, hence this model is called task-shared. In the task-specific variant, there are task-specific layers on top of shared layers, that attend to the specificity of different tasks.\n- **Finetuning vs. direct mapping:** In direct mapping, we used a refurbish module $h$ in front of the pretrained model $f$. The model learns to directly map the amputee inputs $X^{amp}$ to the target output $y^{amp}$. During learning, the pretrained model $f$ is frozen, whereas the refurbish module $h$ is tunable. In finetuning, no refurbish module is used, and the pretrained model $f$ is finetuned to learn a mapping from amputee inputs $X^{amp}$ to target output $y^{amp}$.\n- **Cross-mapping:** Cross-mapping refers to zero-shot transfer, where the pretrained model $f$ is used to predict output trajectories from amputee inputs without any training.  Since no training was involved, it was not possible to report scores for cross-mapping with different training ratios. We acknowledge your concern and will clarify this in the caption of Table 1.\n\nWe hope that answers your questions. We are very happy to elaborate if you need additional details or have other questions. \n\n**Network inversion vs. neighbor search:** As you rightly mentioned, the numerical results show that network inversion and neighbor search exhibit similar performance, with network inversion being slightly higher. The main difference between these methods is that neighbor search requires access to the entire corpus of able-bodied inputs used for training the foundation module during template searching. In contrast, network inversion eliminates this requirement by computing the correction templates directly from the foundation model through gradient optimization. Consequently, the memory requirement for the neighbor-based approach is $ O(N.D)$ due to the need to store the able-bodied dataset (where $N$ is the number of training samples and $D$ is the input dimension) whereas that of network inversion is $O(D)$ as it only requires storage for the current input and gradients while offering slightly higher performance benefits. \n\n**Reporting correction-based results:** Thank you for the excellent suggestion. We initially planned to include the correction-based results as well in Fig. 4. However, since it made the figure cluttered, we decided against it. In the below table, we show the correction-based results for different training ratios. It can be observed that for small amount of training data, simply the correction-based method does not work well. However, as the amount of training data, the correction-based performance also increases (as the refurbish module becomes increasingly accurate). These results show that the performance of the correction-based mapping depends strongly on the accuracy of the refurbish module. By adding the target-based loss, this constraint is relaxed and the model now depends more on the performance of the pretrained model. \n|       Train size      |    0.1    |    0.2    |    0.3    |    0.4    |    0.5    |    0.6    |    0.7    |    0.8    |    0.9    |\n|:---------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|      Output $R^2$     | 0.18±0.14 | 0.41±0.12 | 0.48±0.11 | 0.58±0.11 | 0.57±0.11 | 0.57±0.16 | 0.66±0.09 | 0.63±0.07 | 0.57±0.17 |\n| Refurbish model $R^2$ | 0.20±0.06 | 0.38±0.05 | 0.42±0.06 | 0.46±0.05 | 0.45±0.05 | 0.48±0.06 | 0.48±0.09 | 0.48±0.06 | 0.45±0.09 |\n\n**Finetuning vs reprogramming:** We found that the performance of reprogramming is better compared to finetuning for very low data regimes. A similar observation was also reported in a related study (pointed to by reviewer Du29) that uses input adaptation for GAN conditioning. In terms of tunable parameters, during finetuning, there are \\~500k tunable parameters whereas during reprogramming there are \\~45k tunable parameters.\n\n**Calculation of target data for amputees:** Thank you for the comment and for allowing us to clarify this further. The target is computed from able-bodied individuals with similar anthropometry and walking speeds. \n$$\\tilde{y}\\_k^{amp}=\\frac{(\\sum\\_{(s\\in S_{anthropometric})}\\sum\\_{(n\\in N_{speed})}y\\_k^{(s,n)} )}{(|S\\_{anthropometric} ||N\\_{speed} |)}$$\nwhere $S_{anthropometric}$ is the set of subjects with similar anthropometry as the amputee (height: ±5cm, weight: ±5kg) and $N_{speed}$ is the set of gait cycles where the able-bodied subjects in walked at similar speeds (±0.1m/s) as that of the amputee. The reference timepoint for computing a matching amputee output was based on the phase of the gait cycle.\n\n**Future work:** Our future efforts will expand our approach to a wider range of motion conditions, additional neuromuscular impairments, and scenarios characterized by a scarcity of data. Additionally, we plan to validate our method for real-time applications and for controlling assistive devices, ensuring that our models can be effectively integrated into the real world. We will include a more detailed future work description in our final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Ethical issues were raised with this paper in relation with the disclose of some information but this perceived differently by the two ethical reviewers; R-A4pt sticked to rigour and the letter of the law, whereas R-dHgy opted for trusting and sticking to the spirit of the law. I aligned with R-dHgy but made explicit to the authors that R-A4pt has an important point.
Other than that, this was one highly entertaining discussion (28 replies!) with a fascinating exchange of point of views between the authors and the reviewers; and you got both ends; reviewers sticking to their principles (R-A4pt) whilst others acknowledging the points made in the rebuttal and hence raising the score (R-wD18).
Anyhow, the paper was well appreciated by all reviewers who emphasized the significant impact (R-rCTE) and meaningful choices -of vars and baselines- (R-2Em3), and all acknowledged a strong methodological aspect. I believe, one of the big selling points of this paper and which makes the discussion so rich is that the authors succeeded in making it accessible. Even if one is not an expert on the field it is relatively easy to follow what’s going on.
Of course, the paper had limitations hence why its score was not higher as it could have been; limitations and concerns over generailizability and ack of ground truth (R-rCTE), or the lack of external results to compare to (R-2Em3) were detrimental to the overall appreciation.
Decision: Accept (poster)