Safe Robot Learning in Assistive Devices through Neural Network Repair
Authors: Keyvan Majd
Geoffrey Mitchell Clark
Tanmay Khandait
Siyu Zhou
Sriram Sankaranarayanan
Georgios Fainekos
Heni Amor
Venue: CoRL 2022 Poster (2022)
Keywords: Imitation Learning
Assistive Robotics
Safety
Prosthesis

Abstract: Assistive robotic devices are a particularly promising field of application for neural networks (NN) due to the need for personalization and hard-to-model human-machine interaction dynamics. However, NN based estimators and controllers may produce potentially unsafe outputs over previously unseen data points. In this paper, we introduce an algorithm for updating NN control policies to satisfy a given set of formal safety constraints, while also optimizing the original loss function.  Given a set of mixed-integer linear constraints, we define the NN repair problem as a Mixed Integer Quadratic Program (MIQP). In extensive experiments, we demonstrate the efficacy of our repair method in generating safe policies for a lower-leg prosthesis.

Safe Robot Learning in Assistive Devices through Neural Network Repair
Keyvan Majd"t, Geoffrey Clark!, Tanmay Khandait', Siyu Zhou',
Sriram Sankaranarayanan', Georgios Fainekos", Heni Ben Amor'
Arizona State University, Universityy of Colorado Boulder, 3 Toyota NA-R&D
mm judadddsuuc.m edu
Assistive robotic devices a particularly promising field of applica- tion for neural networks (NN) due to the need for personalization and hard-to- model human-machine interaction dynamics. However, NN based estimators and controllers may produce potentially unsafe outputs over previously unseen data points. In this paper, we introduce an algorithm for updating NN control policies to satisfy given set of formal safety constraints, while also optimizing the orig- inal loss function. Given set of mixed-integer linear constraints. we define the NN repair problem as Mixed Integer Quadratic Program (MIQP). In extensive experiments, we demonstrate the efficacy of our repair method in generating safe policies for a lower-leg prosthesis.
Abstract:
Keywords: Imitation Learning, Assistive Robotics, Safety, Prosthesis
Introduction
Robot learning has the potential to revolutionize the field of wearable robotic devices, such as prosthetics, orthoses and exoskeletons [1]. Often, such devices are designed in "one size fits all" manner using models based on av- erage population statistics. However, machine learning techniques can help adapt control parameters automat- ically to the wearer's individual characteristics, motion patterns or biomechanies. The result a substantial im- provement in ergonomic comfort and quality-of-life. De- spite these benefits, the adoption of machine learning and, in particular, deep learning [2] in this field is still limited, largely due to safety concerns. sis, for example, it may be important to guarantee that the generated control values do not exceed a maximum threshold under set of testing conditions. Other safety constraints include limits on velocities, joint angles and bounds on the change in control inputs.
Controller
NNRapl
Data & Weights
the case of a prosthe-
A lower-leg prosthesis run- ning a neural network for control. A neural network repair process ensures that safety constraints are satisfied.
Figure 1:
In this paper, we derive an algorithm for training neural networks controllers to satisfy given safety specifications, in addition to fitting the given training/test data. In particular, we approach to derive controllers for a robotic lower-leg prosthesis that satisfy basic safety conditions, (see Fig. I). Our proposed approach inputs trained network (e.g., network obtained using backpropagation the training data) along with a specification that places restrictions on the possible outputs for a given set of inputs. It then generates modified set of weights that obeys the desired safety constraints the output using deterministic global optimization. We provide theoretical guarantees of optimality for this technique (assuming numerical errors). Furthermore, in real-world robot experiments we show that the introduced methodology produces safe neural policies for a lower-leg prosthesis satisfying variety of constraints.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Control Networkk
Repairod hichwork
NNRapLayer
Global Constraint
Control race (60g)/
Violated regions
Imput- oulput Constrant
Concitional Comttrant
75
150
225
300
75
150 [s]
225
[s]
Figure 2: Overview of our approach. Left: a given (unsafe) neural network is trained to control prosthesis. However, its outputs violate the formal safety constraints. Right: Using our NN Repair strategy, we identify an adjusted set of neural network weights that removes all violations while still maintaining the underlying behavior of the controller.
Contributions. The proposed method can repair any layer in a network increasing the size of the solution space and the likelihood of a feasible successful repair. More importantly, it comes with theoretical guarantees on successfully repairing all discovered unsafe samples. When compared to retraining or fine-tuning methods, it also has two distinct benefits: (1) it does not require modification of the training data so as to satisfy the constraints, and (2) it does not utilize gradient optimization methods which do not guarantee constraint satisfaction even for the discovered unsafe data points. Finally, when compared to other property driven repair works, i.e., [3, 4, 5, 6, 7], it is applied for the first time to real physical system, i.c., powered prosthetic device. opposed to a model.
Related Works Learning based control for prosthetics is motivated by the challenges in model- ing prosthesss dynamics, which exhibits time varying behavior. Several different approaches that utilize learning based methods for controlling prosthetic devices have been proposed in the lit- erature [8, 9]. Nevertheless, this short review focuses only on neural network (NN) based methods. Gao et al. [10] provide prosthetic controller based on recurrent neural networks, and show that this type of controller effectively minimizes the difference between desired and actual trajectories on a powered prosthetic device. In another direction, Keleş and Yucesoy [11] and Vonsevych et al. [12] focus on utilizing electromyography (EMG) signals to predict control parameters for prosthetic ankles and hands. Our work follows a similar approach in predicting ankle control parameters with neural network, which then drives the prosthesis through a PD controller. The verification of neural networks has been widely studied in order to check specifications for a given NN as a standalone component [13, 14, 15, 16] (for a survey see [17]). or as part of closed loop system [18, 19, 20). Testing techniques can produce counterexamples (adversarial samples) for a variety of NN-based applications, c.g., [21, 14, 22, 23, 24, 25J. One approach to ensuring that a NN satisfics given set of safety properties is to use retraining and fine-tuning based on counter-examplee [26, 27, 7]. However, this approach has a number of pitfalls. First, the labels for the counterexamples need to be available. This may involve further data-colledte cote-coolection, which is often cumbersome. Also, gradient descent opti- mization approaches cannot provide any guarantees that the result satisfies the provided constraints. Our approach, in contrast, avoids generating adversarial examples or performing gradient descent. The problem of "repairing" a network, which involves modifying weights of the network in a mini- mal manner satisfy some safety properties, is thus more involved than simply retraining network on some additional data involving counterexamples. Some methods in the literature [3, 4, 5, 6) make progress toward the goal of repairing NNs. Goldberger et al. [3] can only repair the output layer, which drastically reduces the space of possible successful repairs (and in many cases repair is not even possible). The method of Fu and Li [4] produces patches for certain partitions of the input space wherein the corresponding outputs are modified. The direction of adding patches on the NN output is very promising; however, it is unclear how the approach will scale for high dimensional inputs, since it requires partitioning the input space into affine subregions. Authors in [6] repair the linear regions related to each faulty sample the faulty network's weight space using a decoupled DNN architecture. However, this method causes the repaired network to be discontinuous. There- fore, it cannot be employed robot learning and control applications which the target application of this paper. This method is not also applicable for the networks with more than three inputs.

---

2 Problem Formulation
Without loss of generality, we motivate and discuss our approach using the task of learning safe robot controllers for a lower-leg prosthesis. The goal of this task is to learn a policy Te which generates control values for the ankle angle of the powered prosthesis given a set of sensor values. Most critically, however, policy me is required to satisfy a set of safety constraints V/ Fig. 2 provides an overview of our methodology in addressing this challenge. For now, we assume that an unsafe prior policy network may exist, see Fig. 2 (left). The network parameters @ may be learned with through imitation learning [10], reinforcement learning [28] or any of the common machine learning paradigms. The policy may be optimized for task efficiency, e.g.. stable and low-effort walking gaits, but may not yet satisfy any safety constraints. Our goal is to find an adjusted set of network parameters that satisfy any such constraint. We may now choose, for example, to restrict the control rate to specific bounds. Running the network on the input values of a validation data set reveals that violations occur number of time steps, as seen in Fig. 2. Our approach, called NNRepLayer (Layer-wise Neural Network Repair), takes the original network parameters 0 and the predicates y and yields the updated parameters that generate no violations.
Notation. We denote the set of variables {a1.az,-. a2 .xx with {an) Let xo be a network policy with L hidden layers. The nodes at each layer E 446- are represented by x', where |ae| denotes the dimension of layer (x." represents the input of network). The network's output is also denoted by y x=())). We consider fully connected policy networks with weight and bias terms {(Ow, 0z The training data set of N inputs x and target outputs in denoted by {(=2: (n))== sampled from the input-output space X' x T S Rla*I x Rll We use of to denote the vector of nodes at layer / for sample n. In this work, we focus on the policy networks with the Rectified Linear Unit (ReLU) activation function R(z) = max 23. Thus, given the n sample, in the 1hh hidden layer, we have x' = R (0 + 8))- An activation function is not applied to the last layer, i.e. y OL+12* + 0g.
Problem Statement (Repair Problem). Let To be a trained policy network over the training input- output space X' x T C Rl" x R!lll and (y,x²) be predicate on the output of network for a set of inputs of interest 00 E X S X. The Repair Problem is modify the weight 6n and bias terms 0, of Te such that the repaired policy no. satisfies (yy z.).
The repair of the policy network should not only satisfy the predicate (y x9) but should also maintain the performance of original policy. To satisfy the latter, the method proposed in [3] ensures the satisfaction of predicate only with a minimal deviation of weights in the last layer. However, as we show latter in the experimental results, the repair of last layer is not necessarily feasible or sufficient satisfy the predicates with a minimal deviation from the original parameters. Moreover, the minimal deviation from the original weights is not a sufficient guarantee to maintain the original performance of network. It is well-known that subtle changes in the weights may cause the network to significantly deviate from its original performance [29]. Therefore, it is important for the repaired policy To, to also minimize the loss w.r.t. its original training data. We propose NNRepLayer (Layer-wise Neural Network Repair) that satisfies a predicate y (,, x by repairing a specific layer of the policy network while minimizing the training loss.
3 NNRepLayer
We can formulate our framework as the minimization problem of the loss function E(Ow,, 0b) subject to (x",t) t) E X' x T and 4 (),,z)) for x² E Xr. However, the resulting optimization problem non-convex and difficult to solve due the nonlinear ReLU activation function and high-order nonlinear constraints resulted from the multiplication of terms involving the weight/bias variables. In our approach, we obtain a sub-optimal solution by just focusing on repairing a single layer. We therefore modify the weight and bias terms of a single layer to adjust the predictions so as to minimize E(A,... 9,) and to satisfy (y, ))). Thus, we solve the following problem
Let Th denote a trained policy network with L hidden layers over the training input- output space X x T C TRla0 I x M and V(y, xD) denote a predicate representing constraints on the output of Na for the set of inputs of interest x. E X2 C X. NNRepLayer modifies the weights of
Problem
3

---

layer 1 E f1...., ,L+1 1) in Th such that the new network тe, satisfies V (yy, while minimizing the loss of network FO(O
w. 0%) with respect to its original training set.
Since X, and X' are not necessarily convex, we formulate NNRepLayer over data set [(2% (n)
A' x TUU Xr T, where T is the set of original target values of inputs in X,,- The predicate (x",, y) defined over z. E Xr is not necessarily compatible with the target values in T. It means that the predicate may bound the NN output for X, input space such that not allowing an input x. E X, to reach its target value in T. It is a natural constraint in many applications. For instance, due to the safety constraints, we may not allow a NN controller to follow its original control reference for given unsafe set of input states. For a given layer I, we also define E'OO,.. %%) tz 1.. where Il-I12 denotes the Euclidean norm. Here, since we only repair the weight and bias terms of target layer 1, the loss term E is a function of tf, and 6/, respectively. Hence, the weight and bias terms of all layers except the target layer I are fixed. We define our optimization formulation as follows.
N' in the form of sum of square loss E(0 0)) >m= Ily,, (22, 0,. 03)
NNRepLayer Optimization Formulation. Let Te be neural network with L hidden layers, [=)))==111 (g), z²) be a predicate, and {(a%, (n) J==1 be an input-output data set sampled from (X x T) U (X, x ') over the sets X, X, T, and T all as defined in Problem 1. NNRepLayer minimizes the loss (1) by modifying 0y and 0j subject to the constraints (2)-(5).
Here, constraint (2) represents the lin- ear forward pass of network's ' last layer. Constraint (3) represents the forward pass of hidden layers starting from the layer 1. Except the weight and bias terms of the 1th layer, i.e. 8%, and fj. the weight and bias terms of the subsequent lay-
(1)
E(O 06) + 5,
of Mo
s.l.
(2)
yn =0L+1 2 + 0ft11
ers {(Om.: 0b) ())ee i=1+1 are fixed. The sam- ple values of - are obtained by the weighted sum of the nodes in its previous
(3)
I = R(0 - + 0%)
for (
(4)
v (ynn xh),
for 2 E X2
layers starting from = for all N samples {n), Each ReLU node x' is formu- lated using Big-M formulation [30, 31] of 2 0y 0y
(5)
8 2 110% - @l.init |loo, 110,
8f.mit 1l 2 0.
8%. x' < (0t,a11 + o)) 1b(1 x), and x S ub Φ. where x' E [0, 2o), and
oe E f0, 1) determines the activation status of node x'. The bounds 1b, ub E R are known as Big-M coefficients, 0y - 0g E [lb, ub). that need be tight possible to improve the perfor- mance of MIQP solver. We used Interval Arithmetic (IA) Method [32, 14] to obtain tight bounds for ReLU nodes (read the supplementary materials, Sec. A, for further details on IA). Constraint (4) is a given predicate on y defined over x" E Xz. NNReplayer addresses the predicates of the form Ve vir(a,, y) where C' represents the number of disjunctive propositions and t/ is an affine function of zd and Finally, constraint (5) bounds the entry-wise max-norm еттоr between the weight and bias terms O', and 0/, and the original Of.init and 0% 0f.init by 8. Considering the quadratic loss function E((O!.. o.) and the affine disjunctive forms of V (V 22) and R(0 + +0)), we solve NNRepLayer as a Mixed Integer Quadratic Program (MIQP).
Theorem 1. Given the predicate V (y,,,),, and the input- output data set {(x tn WN sampled from (X T) U (X, x T) over the sets X, Xx, T, and T as defined in Problem 1, assume that 6' and 0% are feasible solutions to (1)-15) Then, V (TT, (2)), is satisfied for all input samples ah.
Proof. Since the feasible solutions o' and 0f satisfy the hard constraint (4) for the repair data set ((ch (n) (TT,, (2²),, x²h) is satisfied.
Given Thm. 1, the following Corollary is straightforward.
Corollary 1. Given the predicate (y x²), and the input-output data set {(2 tn |NN sampled from (Xx XT)TUXXX x xT) over the sets X, X. T, and i as defined in Problem 1, assume that 8 and o5 are the optimal solutions the NNRepLayer. Then, for all input samples 2.9, from {(ag, t,.) 1 N V (TT,(a%))
(4%),, 2%) is satisfied.

---

4 Evaluation
We explore the applicability of the framework in satisfying the following three types of constraints.
Global constraints that encode global bounds on the network's output, i..,, E [Vmmens Vmaz]
Input-output constraints that ensure the network's 's output y to stay within
a certain bound with respect to the network's ' input x². i.e., (1e (x".y) I
oo where C' is the number of constraints and de is an affine function of x. and Finally, conditional constraints that encode if-then-else con- straints described as (vo(4") .y) I O, if x², y E SoJE- where C' specifies the number of conditions, vee is an affine function of x and y, and S. S X x T. We designed a number of experiments to validate that our repair framework can successfully apply these constraints the policy network. Following our motivation, all experiments were performed on the prosthetic walking gait generation task introduced in Fig. 2. Through these experiments aim to answer the following questions: (1) Does our method enable the Prosthetic device to address all the three types of aforementioned constraints? (2) Can device model.
oop Cka
Figure 3: Prosthetic
the repaired controller be employed in a real walking scenario successfully? (3) How robust is the policy repaired through our technique against the unseen constraint constraint-violating samples?
Experimental Setup. In our experiment, we train a policy network Ta for controlling a prosthesis, which then undergoes the repair process to ensure compliance with the safety constraints. To this end, we first train the model using an imitation learning [33] strategy. For data collection, we con- ducted study approved by the Institutional Review Board (IRB), in which we recorded the walking gait of healthy subject without any prosthesis. Walking data included three inertial measurement units (IMUs) mounted via straps to the upper leg (Femur), lower leg (Shin), and foot. The IMUs acquired both the angle and angular velocity of each limb portion in the world coordinate frame at 100Hz. Ankle angle Ola was calculated as a post process from the foot and lower limb IMUs. We then trained the NN to generate the ankle angle from upper and lower limb IMU sensor values. More specifically, the NN model receives the angle and velocity from the upper and lower limb sensors (network inputs z²), aut, å'l, ag, dit respectively, to predict the ankle angle a, (network output y) which is, later, used as the control parameter for a PD controller on the prosthetic. See Fig. 3 for a visualization of the individual sensor readings. We used a sliding window of input variables, denoted as dt (dt- 10 in all our experiments), to account for the temporal influence on the control parameter and to accommodate for noise in the sensor readings. Therefore, the input to the network is dt x |z2%, or more specifically the current and previous dt sensor readings. In all experiments, trained a three-hidden-layer deep policy network with 32 ReLU nodes at each hidden layer. After the networks were fully trained we assessed the policy for constraint violations and collected sam- ples for NNRepLayer. We tested NNRepl policy to satisfy the constraints with subset of the original training data including both adversarial and hon-adversante
on the last and the second to the last layer of network
samples. In all experiments, we used 150 samples in NNRepLayer and a held out set of size 2000 for testing. Finally, the repaired policies to satisfy global and input-output con- straints are tested on a prosthetic device for 10 minutes of walking, see Fig. 5. More specifically, the same healthy subject was fitted with an ankle bypass; a carbon fiber structure molded to the lower limb and constructed such that a prosthetic ankle can be attached to allow the able- bodied subject to walk on the prosthesis, as shown in Fig. 2. The extra weight and off-axis positioning of the device incline the individual towards slower, asymmetrical gaits that generates strides out of the original training distribution [34, 10). The participant then asked to walk again for minutes to assess whether constraints are satisfied.
To evaluate if the repair can be generalized to the unseen adversarial samples, we analyze the vi- olation degree. The violation degree is measured as the distance of the network output with re- spect the constraint set. For each experiment, we explain how this distance between the out- put and the constraint set is calculated. We compared our framework with retraining, fine-tuning [26, 35, 27, 7], and the patch-based repair method in [4] (REASSURE). Adversarial samples in the repair data set are hand-labeled for fine-tuning and retraining so that the target outputs satisfy the given predicates.
fine-tuning, as proposed in [26, 35], we used the collected adversarial data set to train all the parameters of the original policy by gradient descent using a small learn- ing rate (10-4). To avoid over-fitting to the adversarial data set, we trained the weights of the top layer first, and thereafter fine-tuned the remaining layers for few epochs. The same hand-labeling
5

---

8.8
Contral enorr [sog] Control [deg]
Ref.
(c)
Original
Fine-tuned
Violation degree
-10
Retrained
Our method layer
4.4
10.0
Our method mid layen
Repaired REASSURE
6.7-
Violated region
3.3
0.0
3
12 0.0
0.5
1.0
1.5
2.0
Time [s]
L-ddistacc the neighbor
Figure 4: Global constraint: (a) ankle angle, Cta (b) the error between the predicted and the reference controls, (c) the violation degree vs. La-distance between the test and repair sample inputs.
strategy is applied in retraining, except that new policy is trained from scratch for all original training samples. In both methods, we trained the policy until all the adversarial samples in the repair data set satisfy the given predicates on the network's output. Our code is available on GitHub: https: //g.tgtttttt thub. com/k k ma htw.Reepaaaarrrr....mmmrmrr/aa.. git..
4.1 Experiments and Results
Global Constraint. The global constraint ensures that the prosthesis control, i.c., Oz., stays within a certain range and never outputs an unexpected large value that disturbs the user's walking balance. Additionally, the prosthetic device in these scenarios contains a parallel compliant mechanism. As such, either the human subject or the robotic controller could potentially drive the mechanism into the hard limits, potentially damaging the device. In our walking tests, we therefore specified global constraints such that the ankle angle stays within the bounds
utilized
Rep. global constraint
Rep. impar. astput constraint
Cantrol [Aag]]
05
Contoll ros [dg//s]
of [-14, 24] [deg] regardless of whether it is
driven by the human or the robot. In simula- tion experiments, we enforced artificially strict bounds on the ankle angle da to never ex- ceed Og 10 [deg] which is a harder bound to satisfy. We defined the degree of violation as Qa [aminn a0a and min(la, ammar loa axd"") [). otherwise. As shown in Fig. (a)- the repaired network successfully satisfies the constraints in the original faulty regions while maintaining the tracking perfor- mance of the controller in the unconstrained regions. Figure 4 (c) demonstrates that the violation degree stays almost zero for even distant originally violating data points after repairing the mid layer. The red control signal in Fig. also shows that method successfully imposes the control bounds [-14, 24] to the actual prosthesis walking test.
0.0
5.9
11.0
17.6 23.5
Time H4
Figure 5: Real prosthesis walking test results for imposing the global constraint of [-14, 24] to the control (shown in red) and bounding the control rate by 2 [deg/s] (shown in black). The color bar represents the normalized Ly-distance of each test input to its nearest neighbor in the repair set.
Input-output Constraint. Deep neural networks as highly non-linear function approximators have the ability to change the outputs more rapidly than what is feasible for the robotic prosthesis or for the human subject to accommodate. Therefore, possible change of control actions from one time-step to the next. This constraint should act to both smooth the control action in the presence of sensor noise, well as to reduce hard peaks and oscil- lations in the control action. To capture this constraint as an input-output relationship, we trained the policy network by adding the previous d/ control actions ( di as inputs to the policy net- work along the values of upper and lower limb sensors. Imposed constraints in this example follow the form (t) an (1 1)] & Ao prosthetic walking tests, we bounded the control rate by
propose additional constraint over the
Aomaz = 2 [deg/s], and in our simulations we tested An = 1.5 and An = 2 [deg/s]. Our

---

(+)
Control bound
(b) Control bound 1.5
20
Contral rate (dog/ Control [dleg]
10
-20
-2
-4-
0
2
Time [s]
Time [+]
Ref. Orig. predictions
Rep. prodictions REASSURE
Rep. predictions mid layer
Violated region
Figure 6: Input-output constraint: Ankle angles and Ankle angle rates for bounds (a) Doa 2 and
(b) Aa. = 1.5
simulation results in Fig. 6 demonstrate that NNRepl ayer satisfies both bounds on the control rate which subsequently results in a smoother control output. It can also observed that NNRepl.aye Layer successfully preserves the tracking performance of controller. In this experiment, we defined the violation degree as 0 if the (ta I A Aazazz and (ta(t) - Aa otherwise. Figure 7 (a) demonstrates that the violation degree of NNRepLayer is almost zero by increasing the distance of the test samples from the repair set. Same results are obtained in the actual prosthetic walking tests shown in Fig. 5. The satisfaction of input-output constraints (black signal) is achieved even for the samples that are distant from the repair set. Finally, in this experiment, applying NNRepLayer the last layer does not obtain even feasible solution to the optimization problem (1)-(5).
Conditional Constraint. Depending on the ergonomic needs and medical history of a patient,
the attending orthopedic doctor prosthetist may iden- tify certain body configurations that are harmful, e.g. they may increase the risk of osteoarthritis loskeletal conditions [36, 37]. Following this rationale, define region S of joint angles space that should be avoided. An example of such a region is demon- strated in Fig. 8 grey box S ((Ona; (ta) Ond -0.5, 0ta E []] in the joint space of ankle and femur angles. To satisfy this constraint the control rate should be tuned such that the joint ankle and femur an- gles stay out of set S. This constraint can be defined as an if-then-else proposition au E [-2,
1.1
(a)
Violation degree
muscu-
0.7
0.4
0.0
10
(b)
Violation degree
Original
0.7
Repaired
Fine-tuned
0.3
Retrained
(an E (-o:, 1)) V (a,, E [3, x)) which can be for- mulated as the disjunction of linear inequalities network's output. For each given test input and its cor-
0.0
the
0.0
0.7
1.3
2.0
L distance to nearest neighbor
responding output ag. the degree of violation is defined as the distance of aa to the box if aa is outside the box, and 0 otherwise. Figure 8 demonstrates the output of new policy after repairing with NNRepLayer. As it shown, our method avoids the joint ankle and femur angles to enter the unsafe region S. Figure (b) also illustrates low output violation degree for the distant test input samples from the repair input set. Finally, we observed that repairing the last layer does not result in a feasible solution.
Figure The violation degree vs. L2- distance between the test and repair sam- ple inputs for (a) input-output constraint, and (b) conditional constraint cases.
Comparison w\ Fine-tuning, Retraining, and REASSURE. In each experiment, we demon- strated the violation degree and the control signals of our method compared with fine-tuning, retraining [26, 35, 27, 7], and REASSURE [4]. Comparing to [4], while REASSURE guar- antees the satisfaction of constraints in the local repaired linear regions, we showed that this method significantly reduces the performance of network in the repaired local regions, see Fig- ures 4 and 6. This method cannot address the input-output constraints given the faulty sam- ples, and introduces 500 times more faulty samples compared our technique. REASSURE cannot also accommodate the conditional constraints. Unlike REASSURE that guarantees the
7

---

Table 1: The table reports: RT: runtime, MAE: Mean Absolute Error between the repaired and the original outputs, RE: the percentage of adversarial samples that are repaired (Repair Efficacy), and IB: the percentage of test samples that were originally safe but became faulty after the repair (Introduced Bugs). The metrics are the average of 50 runs.
NNRepLayer
REASSURE [4]
RT [8]
MAE
RE [%]
IB [%]]
RT [S]
MAR
RE [%]]
IB [%]]
Global
233 + 159
0.11
90 +
0.09 1 0.20
14
2.3 0.78
97 +
Input-output
112 122
0.5 0.03
98 =
0.19 4: 0.18
30 = 8
0.6 0.03
19 + 4
85
Conditional
480 +: 110
0.35 + 0.07
93 4:
0.11 4: 0.26
Infeasible
Infeasible
Infeasible
Infeasible
Fine-tune
Retrain
RT [S]
MAE
RE [%k]
[%]]
RT [s]
MAE
RE [%]
IB [%]
Global
25
1.2 = 0.03
97 +
0.95 0.45
127
1.4 0.08
98 +
0.65 + 0.40
Input-ouiput
8
0.6 4 0.03
88 d:
2.47 : 0.49
101 +
0.5 +: 0.04
98 d:
0.28 d: 0.32
Conditional
18 3
0.7 + 0.10
72 5
0.27 + 0.25
180 +
0.31 + 0.03
76 2
0.12 +: 0.35
satisfaction of constraints for the samples in the same linear region as the repaired samples, our technique only guarantees the satisfaction of constraints for the repaired samples. While we empirically showed the generalizability of our technique in a local neighborhood of the re- paired samples, our method does not theoretically guarantee the satisfaction of constraints for the unseen adversarial samples. We proposed a sound algorithm in the supplementary materi- als, Sec. B, that guarantees the safety for all other unseen samples. Table better illustrates the success of our method in satisfying the constraints while maintaining the control performance. As shown in Table 1, retraining and NNRepLayer both per- form well in maintaining the minimum absolute error and the generalization of constraint satisfaction to the unseen testing
4.0
samples for global and input-output constraints. However, the
0.6
satisfaction of if-then-else constraints is challenging for re- training and fine-tuning the repair efficacy is dropped by al- most 30% using these techniques. It also highlights the power of our technique in generalizing the satisfaction of conditional constraints to the unseen cases. For further details on the com- parison results, read the supplementary materials, Sec. C.
Ankle angle [deg]
-2.8
-6.1
Original
Repaired
-9.5 -3.0
5 Conclusion & Discussion
-1.6
-0.2
1.2
2.6
Femur angle [deg]
In this paper, we introduced an algorithm for training safe neu- ral network controllers that satisfy formal set of safety con- straints. Our approach, NNRepLayer, performs a global opti- mization step in order perform layer-wise repair of neural network weights. In real-robot experiments, we have shown that the introduced methodology pro- duces safe neural policies for a lower-leg prosthesis satisfying a variety of constraints. We argue that this type of approach is critical for human-centric and safety-critical applications of robot learning, e.g., the next-generation of assistive robotics.
Figure 8: Enforcing the conditional constraints to keep the joint femur- ankle angles out of the grey box.
Discussion. The introduced approach does not generally ensure that for any input the constraints will be satisfied. Instead guarantees this property for all data points provided the time of repair. Hence, proper care has to be taken to ensure that the repair process involves representative samples of the variety of inputs seen in the application domain. From computational vantage point, solving the MIQP underlying NNRepLayer is a demanding process which scales with the size of the net- work. In our experiments. we successfully repaired NN layers with up to 256 neurons, with global optimization taking between multiple minutes and up to 10 hours (read the supplementary materials, Sec. D. for the detailed experimental results). Moreover, we showed the supplementary materials, Sec. E. that the repair of randomly selected sub-nodes of hidden layer can accurately repair the network in much shorter time (more that times faster than the full repair). Finally, our approach limited to repairing individual layers in network. Early results on iteratively repairing multiple lay- ers are promising and will be reported in the future. However, our approach cannot simultaneously repair multiple layers or the entire network.
8

---

Acknowledgments
This work was partially supported by the National Science Foundation under grants CNS-1932068, IIS-1749783, and CNS-1932189.
References
[1] M. Kyrarini, F. Lygerakis, A. Rajavenkataanaryaaaan, C. Sevastopoulos, H. Nambiappan, K. K. Chaitanya, A. R. Babu, J. Mathew, and F. Makedon. A survey of robots in healthcare. Technologies, 9()):8 2021.
[2] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521(7553)::36, 2015.
[3] B. Goldberger, G. Katz, Y. Adi, and J. Keshet. Minimal modifications of deep neural networks using verification. In 23rd International Conference Logic for Programming, Artificial Intelligence and Reasoning, volume 73, pages 268,278, 2020.
[4] F. Fu and W. Li. Sound and complete neural network repair with minimality and locality guarantees. 10th International Conference on Learning Representations (ICLR), 2021.
[5] X. Yang. T. Yamaguchi, H.-D. Tran, B. Hoxha, T. T. Johnson, and D. Prokhorov. Neural network repair with reachability analysis. In International Conference on Formal Modeling and Analysis of Timed Systems, pages 221-236. Springer, 2022.
[6] M. Sotoudeh and A. Thakur. Provable repair of deep neural networks. In 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 588 0038600,2220 2021.
[7] G. Dong, J. Sun, X. Wang, X. Wang, and T. Dai. Towards repairing neural networks correctly. In IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), pages 714-725, 2021.
[8] N. Thatte, N. Srinivasan, and H. Geyer. Real-time reactive trip avoidance for powered trans- femoral prostheses. In Robotics Systems and Sciences (RSS), 2019.
[9] G. Clark, J. Campbell, and B. Amor. Learning predictive models for ergonomic control of prosthetic devices. In Kober, F. Ramos, and C. Tomlin, editors, Conference on Robot Learn- ing, volume 155 of Proceedings of Machine Learning Research, pages 2268-2278. PMLR, 2020.
[10] C. Gao, R. Gehlhar, A. D. Ames, S.-C. Liu, and T. Delbruck. Recurrent neural network con- trol of a hybrid dynamical transfemoral prosthesis with edgedrnn accelerator. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5460-5466. IEEE, 2020.
[11] A. D. Keleş and C. A. Yucesoy. Development of neural network based control algorithm for powered ankle prosthesis. Journal of Biomechanics, 113:1 10087, 2020.
[12] K. Vonsevych, M. Goethel, J. Mrozowski, J. Awrejcewicz, and M. Bezuglyi. Fingers move- ments control system based on artificial neural network model. Radioelectronies and Commu- nications Systems, 62(1):23-33, 2019.
[13] R. Ehlers. Formal verification of piece-wise linear feed-forward neural networks. In 15th In- ternational Symposium on Automated Technology for Verification and Analysis (ATVA), pages 269-286. Springer, 2017.
[14] V. Tjeng, K. Y. Xiao, and Tedrake. Evaluating robustness of neural networks with mixed integer programming. In 7th International Conference on Learning Representutions (ICLR), 2019.
[15] S. Dutta, S. Jha, S. Sankaranarayanan, and A. Tiwari. Output range analysis for deep feed- forward neural networks. In 10th International Symposium NASA Formal Methods (NFM), volume 10811 1 of LNCS, pages 121-138. Springer, 2018.

---

[16] G. Katz, D. A. Huang, D. Ibeling. K. Julian, C. Lazarus, R. Lim, P. Shah, S. Thakoor, H. Wu, A. Zeljié, al. The marabou framework for verification and analysis of deep neural networks. In International Conference on Computer Aided Verification, pages 443-452 Springer, 2019.
[17] C. Liu, T. Amon, C. Lazarus, C. Strong, C. Barrett, and M. J. Kochenderfer. Algorithms for verifying deep neural networks. Foundations and Trends in Optimization, 4(3-4);2444004, 2021. ISSN 2167-3888.
[18] X. Sun, H. Khedr, and Y. Shoukry. Formal verification of neural network controlled au- tonomous systems. In 22nd ACM International Conference on Hybrid Systems: Computation and Control, 2019.
[19] R. Ivanov, T. J. Carpenter, J. Weimer, R. Alur, J. Pappas, and I. Lee. Verifying the safety of autonomous systems with neural network controllers. ACM Transactions on Embedded Computing Systems (TECS), 20(1): 2020.
[20] S. Dutta, S. Jha, S. Sankaranarayanan, and A. Tiwari. Learning and verification of feedback control systems using feedforward neural networks. In Analysis and Design of Hybrid Systems, 2018.
[21] X. Huang, M. Kwiatkowska, S. Wang. and M. Wu. Safety verification of deep neural networks. In R. Majumdar and V. Kunčak, editors, 29th International Conference on Computer Aided Verification (CAV), volume 10426 of LNCS, pages 3-29. Springer, 2017.
[22] Y. Tian, K. Pei, Jana, and B. Ray. Deeptest: Automated testing of deep-neural-networkk driven autonomous cars. In 40th International Conference on Software Engineering (ICSE), 2018.
[23] T. Dreossi, A. Donze, and S. A. Seshia. Compositional falsification of with machine learning components. Journal of Automated Reasoning, 63:1031-1053, 2019.
systems
[24] C. E. Tuncali, G. Fainekos, D. Prokhorov, H. Ito, and J. Kapinski. Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Transactions on Intelligent Vehicles, 5:265-280, 2020. doi:10.1
DD0/TVV.00192955000
[25] T. Dreossi, S. Ghosh, X. Yue, K. Keutzer, A. L. Sangiovanni- Vinceentliiiieenlllli.... and A. Seshia. Counterexample-guidee data augmentation. In Proceedings of the Twenty-Sevnnth Interna- tional Joint Conference on Artificial Intelligence, IJCAI, pages 2071-2078. 2018.
[26] A. Sinitsin, V. Plokhotnyuk, D. Pyrkin, S. Popov, and A. Babenko. Editable neural networks. In International Conference on Learning Representations, 2019.
[27] X. Ren, B. Yu, H. Qi, F. Juefei-Xu, Z. Li, W. Xue, L. Ma, and J. Zhao. Few-shot guided mix for dnn repairing. In 2020 IFFE International Conference on Software Muintenance and Evolution (ICSME), pages 717-721. IEEE, 2020.
[28] X. Gao, J. Si, Y. Wen, M. Li, and H. H. Huang. Knowledge- guided reinforcement learning control for robotic lower limb prosthesis. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 754-760, 2020. 100100.10000CR::444445500000 202.44409995222299944449
[29] X. Zeng and D. Yeung. Sensitivity analysis of multilayer perceptron to input and weight perturbations. IEEE Transactions on Neural Networks, 12(6) 2001. 1oi:10.11099 72.963772.
[30] Belotti, L. Liberti, A. Lodi, G. Nannicini, A. Tramontani, et al. Disjunctive inequalities: applications and extensions. Wiley Encyclopedia of Operations Research and Management Science, 2: 2:1441-1450, 2011.
[31] C. Tsay, Kronqvist, A. Thebelt, and R. Misener. Partition-based formulations for mixed- integer optimization of trained relu neural networks. Advances in Neural Information Process- ing Systems, 34:3068-300 -0000 2021.
[32] R. F. Moore, R. B. Kearfott, and M. J. Cloud. Introduction to interval analysis/ramon e, Moore, R. Baker Kearfou, Michael J. Cloud. Philadelphia, 2009.
10

---

[33] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-22 242, 1999.
[34] R. Cortino, E. Bolivar-Nieto, T. K. Best, and R. D. Gregg- Stair ascent phase-variable control of powered knee-ankle prosthesis. In 2022 International Conference on Robotics and Automation (ICRA), pages 5673-5678. IEEE, 2022.
[35] V. Taormina, D. Cascio, L. Abbene, and G. Raso. Performance fine-tuning convolutional neural networks for hep-2 image classification. Applied Sciences, 10(19):6940, 2020.
[36] D. C. Morgenroth, mechanical perspective. PM&R, 4(5):220- 2012.
Gellhorn, and P. Suri. Ostcoarthritis in the disabled population:
[37] A. Ekizos. A. Santuz, A. Schroll, and A. Arampatzis. The maximum Iyapunov exponent during walking and running: Reliability assessment of different marker-sets. Frontiers Physiology, 9:1101, 2018. ISSN 1664-042X.


Reviews:
[{"review_id": "nest5G2_l3f", "reviewer": "Reviewer_pwYF", "summary": "This paper proposes an algorithm to repair neural networks to ensure they satisfy certain properties.  Given a trained neural network that violates some specification, the paper uses mixed-integer programming to change the weights of some layers to ensure that the resulting network satisfies the specification. The paper provides some numeric examples using a prosthesis walking assistive robot to support their proposed framework.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Applying the repair algorithm to a real-world example like prosthesis walking is interesting.\n\nWeaknesses:\n- Compared to the literature, the proposed method provides weak guarantees. Other algorithms in the literature (which the paper cited only 3 of them) ensure the resulting NN is correct *for all inputs* while the proposed method provides such a guarantee to *the samples available in the dataset*. There is no guarantee that the proposed method will not introduce more counterexamples in areas where the original NN (before the repair) was correct. \n\n- The paper provides no comparison against the other NN repair techniques.\n\n- The MIQP formulation is straightforward and has been extensively used in the literature for NN verification and repair. While other papers in the literature try to provide novel heuristics to better find the solution to these MIQP problems, this paper only applies off-the-shelf solvers. \n\n- The complexity of the proposed MIQP program grows with the number of samples in the dataset. This seems to be why the framework was applied to simplistic settings (150 samples and 64 neurons). Other techniques in the literature do not depend on the dataset size and hence do not suffer from such complexity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "- The novelty of the proposed framework is limited.\n- The theoretical guarantee of the proposed framework is weaker than those in the literature.\n- Lack of comparison with the results in the literature.\n- Algorithmic complexity", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Is the MIQP problem always feasible? How to choose which layer to repair? What happens if the NN can not be repaired by adjusting the weights of a single layer?\n\n- While the numerical analysis tries to check the rate of the adversarial examples before and after the repair, several critical pieces of information are not present. In particular, the experiments should report the fraction of adversarial examples introduced by the repair, i.e., how many non-adversarial inputs became adversarial after the repair. This can be done by evaluating the original NN on the adversarial inputs found in the repaired one to check whether these adversarial examples were introduced by the repair. \n"}}, {"review_id": "ijqQIF3OXpF", "reviewer": "Reviewer_4TJw", "summary": "This paper presents safe-learning-based controllers for a prosthesis device. It proposed NNReplLayer (Layer-wise Neural Network Repair) that modifies the weights and biases of the neural network to avoid violations.\nThis study performed experiments with a real prosthesis and a healthy subject under three different constraints. \nI also compared the proposed method with the fine-tuned and retrained ones. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTH\nThis paper is well written and tackles significant problems for prosthesis devices. The limitations are clearly presented. As this paper describes, few studies consider the safety of learning-based control for prosthesis devices. Meanwhile, it proposes an exciting idea by employing recent network repair techniques.\n\nWEAKNESS\nAlthough this study presents an attractive idea, the impacts of the results look weak. As suggested in the following lists, more explanations or evaluations are required for acceptance.\n\na) My main concern is whether this method is the best for a safe assistive device controller. Recent studies have demonstrated many safe-RL or safe imitation learning approaches (e.g., the review paper might be helpful [r1], but not limited to this paper). Although few cases use safe learning for assistive devices, I wonder if this study's setting can adopt other safe learning techniques.\nHowever, this study lacks discussions about safe learning approaches. Please survey more related works and describe whether the network repair is the best way. Besides, it will be much better to compare the existing safe-learning controllers.\n\n[r1] Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., & Schoellig, A. P. (2021). Safe learning in robotics: From learning-based control to safe reinforcement learning. arXiv preprint arXiv:2108.06266. https://arxiv.org/pdf/2108.06266.pdf\n\nb) The proposed method successfully regulates the control inputs; however, the difference between the proposed and retrained network does not seem significant as the retrained network also showed a small violation degree in global and input-output constraints. \nThis paper needs more explanations or results to describe the significance. For example, how about plotting the other joints or trunk? I would like to see whether the user shows unstable walking when a violation occurs.\nBesides, it would be better to plot the control and control rate in the conditional constraints like Figs. 4 and 7.\n\nc) The horizontal axes of figures 4, 5, and 7 show the time [s]. I suspect they should be timestep as the plot shows a few strides. The user might walk incredibly slowly if the scale is time [s].\n\nd) The term of the title: \"assistive devices,\" would have a broader meaning than this study's scope. I would recommend using the \"prosthesis device\" instead.\n\ne) The proposed section is well and generally described; however, it would be better to explain what the variable (e.g., x, y, and t) stands for in the prosthesis context. The readers will be easier to follow.\n\nf) Figs. 4 and 7 are difficult to see the differences in each condition. Please make them larger.\n\ng) In Fig. 4 (c), the repaired last layer in global constraints showed a higher violation degree. Please provide the reason or discussions for the negative performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is well written, and the core idea is interesting. However, I am concerned about the impact of the results, the relationship to other safe learning, and the benefits in the prosthesis contexts, leading to weak reject recommendations. More explanations and/or comparisons might increase the grade after the rebuttal. \n\nAfter rebuttal comment\n\nThank the authors for addressing my comment. All of my concerns have been solved. I changed my recommendation to weak acceptance. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "As described in the weakness, please provide enough surveys for the safe learning control (or comparisons with the other methods).\nBesides, please clarify the impacts of the results by adding explanations and plots to demonstrate the benefits of safe prosthesis control."}}, {"review_id": "SmtTETB1MEO", "reviewer": "Reviewer_TbCe", "summary": "In order to force neural networks to satisfy certain constraints in safety-critical applications, the authors propose directly modifying weights so that satisfaction of the constraints is formally guaranteed. They achieve this goal by formulating the repair problem as a Mixed Integer Quadratic Program problem and finding an analytic solution. They find that their method outperforms fine-tuning on adherence to a number of constraints, and outperforms retraining the network from scratch on adherence to conditional constraints. They validate this result on a real prosthetic leg.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper introduces an interesting alternative to fine-tuning, which involves using Mixed Integer Quadratic Programming to enforce constraints. I haven't seen any approaches before attempting to solve modify neural nets using MIQP, but the application to safety-critical systems makes a great deal of sense. The main appeal of this method seems to be that it is possible to guarantee that if an answer is returned, then the constraint will be exactly satisfied at all known data points, rather than approximately satisfied, as would be the case for fine-tuning or retraining. This is definitely an appealing quality. \n\nThis just seems like an absolutely crushingly long time to run on a width 32 MLP, such that it's really hard to imagine this approach scaling up to networks of any significant size. Furthermore, because this is solved by an NP complete method, it doesn't seem like there are many ways that clever optimizations could reduce the runtime. I would find this tradeoff more acceptable if the solver was use to produce a guarantee that the neural network can never produce outputs that violate the constraint for all inputs. After all, this is for systems that prioritize safety, so it makes sense to trade off training time for theoretical guarantees. But this method still has to account for generalization, because the guarantee only applies for the training set. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work is interesting, and hopefully offers future avenues into using NNs for safety critical applications. At present the method does not seem usable for any but the smallest networks, and offers only narrow improvements over the presumably-faster method of retraining a network from scratch. However, this niche of small safety-critical networks does have its uses (such as the prosthetic leg mentioned here). For this reason, I recommend acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I understand that given a neural network to repair, NNRepLayer always returns the same set of weights. However, it still depends on the pre-trained network, which is randomly initiallized and whose training is partially stochastic. As such, the experiments really should be repeated across multiple trained networks to measure statistical significance. \n\nIt's also a little unclear what some of the baselines are. I assume \"retrain\"  means \"retrain a new randomly-initialized network from scratch\", but this is not made explicit and leaves some room for ambiguity. "}}, {"review_id": "9w0Hag-XI-Q", "reviewer": "Reviewer_qM6j", "summary": "The authors present a method for updating/repairing a neural network such that the outputs on given examples do not violate safety constraints.  They show that the method is sound and that it produces safer trajectories in a combination of simulated and real-world experiments with prostheses.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Tested on an actual prosthesis\n- Nice mix of empirical and theoretical results\n- This approach seems like it could generalize to other domains as well and may be of interest to robot learning in general\n\n\nWeaknesses\n- The method doesn't actually produce guarantees on unseen examples; it only guarantees that known violations have been repaired.  This leaves it a bit up in the air whether the method would actually result in safe NN controllers in practice\n- I understand that it is normal to evaluate such algorithms with nondisabled participants, but I would at least like to see some discussion on how the results might be different with an actual prosthesis user (e.g., how would this work with someone with whom you can't collect \"correct\" gait data because they are a double lower-limb amputee or because they don't have a \"normal\" gait)\n- the method is very computationally expensive\n\nMinor notes\n- Fig 4 subfigures are not labeled", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper makes an interesting contribution to deep learning for robotics; although expensive, the method is shown to improve over simply re-training a network especially for complex constraints (which often show up in robotics domains).  The approach is validated with simulated and empirical results although not with an actual prosthesis user.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Answer the following questions:\n\nHow is the layer that is repaired chosen?\nCan the computational expense be justified a little more clearly?\nHow might the results be the same (or different) with an actual prosthesis user?\nDiscuss a little more how having representative samples would result in a generally safe controller? Can you be sure that if two examples are close (in some sense) and one of them is repaired, that the second would be repaired too?"}}]

Author rebuttal:

Decision: Accept (poster)