Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In order to force neural networks to satisfy certain constraints in safety-critical applications, the authors propose directly modifying weights so that satisfaction of the constraints is formally guaranteed. They achieve this goal by formulating the repair problem as a Mixed Integer Quadratic Program problem and finding an analytic solution. They find that their method outperforms fine-tuning on adherence to a number of constraints, and outperforms retraining the network from scratch on adherence to conditional constraints. They validate this result on a real prosthetic leg.
strengths_and_weaknesses: The paper introduces an interesting alternative to fine-tuning, which involves using Mixed Integer Quadratic Programming to enforce constraints. I haven't seen any approaches before attempting to solve modify neural nets using MIQP, but the application to safety-critical systems makes a great deal of sense. The main appeal of this method seems to be that it is possible to guarantee that if an answer is returned, then the constraint will be exactly satisfied at all known data points, rather than approximately satisfied, as would be the case for fine-tuning or retraining. This is definitely an appealing quality. 

This just seems like an absolutely crushingly long time to run on a width 32 MLP, such that it's really hard to imagine this approach scaling up to networks of any significant size. Furthermore, because this is solved by an NP complete method, it doesn't seem like there are many ways that clever optimizations could reduce the runtime. I would find this tradeoff more acceptable if the solver was use to produce a guarantee that the neural network can never produce outputs that violate the constraint for all inputs. After all, this is for systems that prioritize safety, so it makes sense to trade off training time for theoretical guarantees. But this method still has to account for generalization, because the guarantee only applies for the training set. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The work is interesting, and hopefully offers future avenues into using NNs for safety critical applications. At present the method does not seem usable for any but the smallest networks, and offers only narrow improvements over the presumably-faster method of retraining a network from scratch. However, this niche of small safety-critical networks does have its uses (such as the prosthetic leg mentioned here). For this reason, I recommend acceptance.
issues: I understand that given a neural network to repair, NNRepLayer always returns the same set of weights. However, it still depends on the pre-trained network, which is randomly initiallized and whose training is partially stochastic. As such, the experiments really should be repeated across multiple trained networks to measure statistical significance. 

It's also a little unclear what some of the baselines are. I assume "retrain"  means "retrain a new randomly-initialized network from scratch", but this is not made explicit and leaves some room for ambiguity.
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes an algorithm to repair neural networks to ensure they satisfy certain properties.  Given a trained neural network that violates some specification, the paper uses mixed-integer programming to change the weights of some layers to ensure that the resulting network satisfies the specification. The paper provides some numeric examples using a prosthesis walking assistive robot to support their proposed framework.
strengths_and_weaknesses: Strengths:
- Applying the repair algorithm to a real-world example like prosthesis walking is interesting.

Weaknesses:
- Compared to the literature, the proposed method provides weak guarantees. Other algorithms in the literature (which the paper cited only 3 of them) ensure the resulting NN is correct *for all inputs* while the proposed method provides such a guarantee to *the samples available in the dataset*. There is no guarantee that the proposed method will not introduce more counterexamples in areas where the original NN (before the repair) was correct. 

- The paper provides no comparison against the other NN repair techniques.

- The MIQP formulation is straightforward and has been extensively used in the literature for NN verification and repair. While other papers in the literature try to provide novel heuristics to better find the solution to these MIQP problems, this paper only applies off-the-shelf solvers. 

- The complexity of the proposed MIQP program grows with the number of samples in the dataset. This seems to be why the framework was applied to simplistic settings (150 samples and 64 neurons). Other techniques in the literature do not depend on the dataset size and hence do not suffer from such complexity.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: - The novelty of the proposed framework is limited.
- The theoretical guarantee of the proposed framework is weaker than those in the literature.
- Lack of comparison with the results in the literature.
- Algorithmic complexity
issues: - Is the MIQP problem always feasible? How to choose which layer to repair? What happens if the NN can not be repaired by adjusting the weights of a single layer?

- While the numerical analysis tries to check the rate of the adversarial examples before and after the repair, several critical pieces of information are not present. In particular, the experiments should report the fraction of adversarial examples introduced by the repair, i.e., how many non-adversarial inputs became adversarial after the repair. This can be done by evaluating the original NN on the adversarial inputs found in the repaired one to check whether these adversarial examples were introduced by the repair.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a method for updating/repairing a neural network such that the outputs on given examples do not violate safety constraints.  They show that the method is sound and that it produces safer trajectories in a combination of simulated and real-world experiments with prostheses.  
strengths_and_weaknesses: Strengths
- Tested on an actual prosthesis
- Nice mix of empirical and theoretical results
- This approach seems like it could generalize to other domains as well and may be of interest to robot learning in general


Weaknesses
- The method doesn't actually produce guarantees on unseen examples; it only guarantees that known violations have been repaired.  This leaves it a bit up in the air whether the method would actually result in safe NN controllers in practice
- I understand that it is normal to evaluate such algorithms with nondisabled participants, but I would at least like to see some discussion on how the results might be different with an actual prosthesis user (e.g., how would this work with someone with whom you can't collect "correct" gait data because they are a double lower-limb amputee or because they don't have a "normal" gait)
- the method is very computationally expensive

Minor notes
- Fig 4 subfigures are not labeled
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper makes an interesting contribution to deep learning for robotics; although expensive, the method is shown to improve over simply re-training a network especially for complex constraints (which often show up in robotics domains).  The approach is validated with simulated and empirical results although not with an actual prosthesis user.
issues: Answer the following questions:

How is the layer that is repaired chosen?
Can the computational expense be justified a little more clearly?
How might the results be the same (or different) with an actual prosthesis user?
Discuss a little more how having representative samples would result in a generally safe controller? Can you be sure that if two examples are close (in some sense) and one of them is repaired, that the second would be repaired too?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents safe-learning-based controllers for a prosthesis device. It proposed NNReplLayer (Layer-wise Neural Network Repair) that modifies the weights and biases of the neural network to avoid violations.
This study performed experiments with a real prosthesis and a healthy subject under three different constraints. 
I also compared the proposed method with the fine-tuned and retrained ones. 
strengths_and_weaknesses: STRENGTH
This paper is well written and tackles significant problems for prosthesis devices. The limitations are clearly presented. As this paper describes, few studies consider the safety of learning-based control for prosthesis devices. Meanwhile, it proposes an exciting idea by employing recent network repair techniques.

WEAKNESS
Although this study presents an attractive idea, the impacts of the results look weak. As suggested in the following lists, more explanations or evaluations are required for acceptance.

a) My main concern is whether this method is the best for a safe assistive device controller. Recent studies have demonstrated many safe-RL or safe imitation learning approaches (e.g., the review paper might be helpful [r1], but not limited to this paper). Although few cases use safe learning for assistive devices, I wonder if this study's setting can adopt other safe learning techniques.
However, this study lacks discussions about safe learning approaches. Please survey more related works and describe whether the network repair is the best way. Besides, it will be much better to compare the existing safe-learning controllers.

[r1] Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., & Schoellig, A. P. (2021). Safe learning in robotics: From learning-based control to safe reinforcement learning. arXiv preprint arXiv:2108.06266. https://arxiv.org/pdf/2108.06266.pdf

b) The proposed method successfully regulates the control inputs; however, the difference between the proposed and retrained network does not seem significant as the retrained network also showed a small violation degree in global and input-output constraints. 
This paper needs more explanations or results to describe the significance. For example, how about plotting the other joints or trunk? I would like to see whether the user shows unstable walking when a violation occurs.
Besides, it would be better to plot the control and control rate in the conditional constraints like Figs. 4 and 7.

c) The horizontal axes of figures 4, 5, and 7 show the time [s]. I suspect they should be timestep as the plot shows a few strides. The user might walk incredibly slowly if the scale is time [s].

d) The term of the title: "assistive devices," would have a broader meaning than this study's scope. I would recommend using the "prosthesis device" instead.

e) The proposed section is well and generally described; however, it would be better to explain what the variable (e.g., x, y, and t) stands for in the prosthesis context. The readers will be easier to follow.

f) Figs. 4 and 7 are difficult to see the differences in each condition. Please make them larger.

g) In Fig. 4 (c), the repaired last layer in global constraints showed a higher violation degree. Please provide the reason or discussions for the negative performance.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper is well written, and the core idea is interesting. However, I am concerned about the impact of the results, the relationship to other safe learning, and the benefits in the prosthesis contexts, leading to weak reject recommendations. More explanations and/or comparisons might increase the grade after the rebuttal. 

After rebuttal comment

Thank the authors for addressing my comment. All of my concerns have been solved. I changed my recommendation to weak acceptance. 
issues: As described in the weakness, please provide enough surveys for the safe learning control (or comparisons with the other methods).
Besides, please clarify the impacts of the results by adding explanations and plots to demonstrate the benefits of safe prosthesis control.
review_title: null

Meta-review: metareview: The reviewers are leaning towards accepting this paper. I appreciate the thorough responses from the authors. I also appreciate the supplementary material which made the paper stronger.

The reviewer pwYF is still against this work. However, I believe that the authors' response is convincing (could be better if it is more polite).

Summary: The reviewer pwYF has a strong opinion against this work. The reviewer made many valid points and the authors should answer them clearly. In general, the reviewers have many common concerns regarding the theoretical guarantees and the computation cost.

Quality: On average, the reviewers rated this paper fair. Some reviewers argued that this paper needs to show more evidence to back up the claims

Clarity: All reviewers rated that this paper is clear.

Originality: The reviewer pwYF mentioned that MIQP has been used for NN verification and repair extensively. The authors should address this comment.

Significance: Overall, the reviewers questioned the impact of this work. Many reviewers questioned the significance or usefulness of the safety guarantee. The method is safe only on the training data and there is no guarantee that it will be safe on the real hardware. The reviewers are concerned about the computation cost of the method. They generally agree that it is not going to be scalable to bigger data sizes.  The reviewer 4TJw mentioned that there are more suitable approaches for safe assistive device control with a reference.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)