Neuromechanical Autoencoders: Learning to Couple Elastic and Neural Network Nonlinearity
Authors: Deniz Oktay
Mehran Mirramezani
Eder Medina
Ryan P Adams
Venue: ICLR 2023 notable top 25% (2023)
Keywords: morphological computation
mechanical metamaterials
computational mechanics
mechanical co-design
automatic differentiation
differentiable simulation

Abstract: Intelligent biological systems are characterized by their embodiment in a complex environment and the intimate interplay between their nervous systems and the nonlinear mechanical properties of their bodies. This coordination, in which the dynamics of the motor system co-evolved to reduce the computational burden on the brain, is referred to as "mechanical intelligence" or "morphological computation". In this work, we seek to develop machine learning analogs of this process, in which we jointly learn the morphology of complex nonlinear elastic solids along with a deep neural network to control it. By using a specialized differentiable simulator of elastic mechanics coupled to conventional deep learning architectures---which we refer to as neuromechanical autoencoders---we are able to learn to perform morphological computation via gradient descent. Key to our approach is the use of mechanical metamaterials---cellular solids, in particular---as the morphological substrate. Just as deep neural networks provide flexible and massively-parametric function approximators for perceptual and control tasks, cellular solid metamaterials are promising as a rich and learnable space for approximating a variety of actuation tasks. In this work we take advantage of these complementary computational concepts to co-design materials and neural network controls to achieve nonintuitive mechanical behavior. We demonstrate in simulation how it is possible to achieve translation, rotation, and shape matching, as well as a "digital MNIST" task. We additionally manufacture and evaluate one of the designs to verify its real-world behavior.


Published as a conference paper at ICLR 2023
NEUROMECHANICAL AUTOENCODERS: LEARNING TO COUPLE ELASTIC AND NEURAL NETWORK NONLINEARITY
Deniz Oktay, Mehran Mirramezani, Eder Medina, Ryan P. Adams Department of Computer Science Princeton University (doktay, mehranmir em2368, ppr)etnnntttoocc edu
ABSTRACT
Intelligent biological systems are characterized by their embodiment in complex environment and the intimate interplay between their nervous systems and the nonlinear mechanical properties of their bodies. This coordination, in which the dynamics of the motor system co-evolved to reduce the computational burden on the brain, is referred to "mechanical intelligence" tion". this work, we seek to develop machine learning analogs of this process, which we jointly learn the morphology of complex nonlinear elastic solids along with a deep neural network to control it. By using a specialized differentiable sim- ulator of elastic mechanies coupled conventional deep learning architectures- which we refer to as neuromechanical autoencoders-ww are able to learn to per- form morphological computation via gradient descent. Key to our approach is the use of mechanical metamaterials cellular solids, in particular as the morpho- ccelullar
"morphological computa-
logical substrate. Just as deep neural networks provide flexible and massively- parametric function approximators for perceptual and control tasks, cellular solid metamaterials are promising as rich and learnable space for approximating a variety of actuation tasks. In this work we take advantage of these complemen- tary computational concepts to co-design materials and neural network controls to achieve nonintuitive mechanical behavior. We demonstrate in simulation how it is possible to achieve translation, rotation, and shape matching, as well as a "digital MNIST" task. We additionally manufacture and evaluate one of the designs verify its real-world behavior.
INTRODUCTION
Mechanical intelligence, or morphological computation (Paul 2006 Hauser et al. 2011) is the idea that the physical dynamics of actuator may interact with a control system to effectively reduce the computational burden of solving the control task. Biological systems perform morphological computation in a variety of ways, from the compliance of digits in primate grasping (Jeannerod 2009 Heinemmnn et al., 2015). to the natural frequencies of legged locomotion (Collins et al.) 2005 Holmes et 2006) Ting & McKay 2007), to dead fish being able to "swim" " in vortices (Beal al,, 2006 Lauder et al. 2007 Eldredge & Pisani 2008). Both early (Sims) 1994) and modern (Gupta cct al. 2021) work have used artificial evolutionary methods to design mechanical intelligence, but has remained difficult to design systems de novo that are comparable to biological systems that have evolved over millions years. We ask:
Can instead learn morphological computation using gradient descent?
Morphological computation requires that a physical system be capable of performing complex tasks using, e.g., elastic deformation. The mechanical system's ' nonlinear properties work in tandem with neural information processing so that challenging motor tasks require less computation. To learn an artificial eechanically-inteligeeee system, must therefore be able to parameterize rich space of mechanisms with the capability of implementing nonlinear physical "functions" that connect input forces displacements to the desired output behaviors. There are various desiderata for such

---

Published as a conference paper at
ICLR 2023
Encoder
Actuation
OOLD
Task Set
Decoder
Figure 1: A schematic depiction neuromechanical autoencoder. A neural encoder is parame- terized by 0, while mechanical decoder has geometry (morphology) parameterized by ø. A task sampled from a distribution and is fed into the neural network encoder. The neural network pro- duces actuations which displace the mechanical structure to perform the task, in this case being shape matching (Section 3.2 Given a task loss, 0 and á are optimized jointly by gradient descent.
mechanical design space: 1) it must contain a wide variety of structures with complex nonlinear elastic deformation patterns; 2) its parameters should be differentiable and of fixed cardinality; and 3) the designs should be easily realizable with standard manufacturing techniques and materials. These characteristics are achieved by mechanical metamaterials.
Metamaterials are structured materials that have properties unavailable from natural materials. Al- though metamaterials are often discussed in the context of electromagnetic phenomena, there is substantial interest in the development mechanical metamaterials in which geometric heterogene- ity achieves unusual macroscopic behavior such negative Poisson's ratio (Bertoldi al. 2010). In biological systems, morphological computation often takes the form of sophisticated nonlinear compliance and deformation, resulting in a physical system that is more robust and casier control for a variety of tasks (Paul 2006; Hauser et al. (2011). This type of behavior is typically not present in off fff-the-shelf robotic systems and is difficult to design a priori. Mechanical metamaterials, on the other hand, offer platform for mechanically-inter intelliggent systems using relatively accessible manu- facturing techniques, such as 3-D printing.
The mechanical metamaterials we explore in this paper are cellular solids: porous structures where different patterns of macroscopic pores can lead different nonlinear deformation behaviors. By constructing a solid with a large number of such pores, and then parameterizing the pore shapes nonuniformly across the solid, it is possible to achieve large design space of nonlinear mechanical structures while nevertheless having a differentiable representation of fixed cardinality. The key to modern machine learning has been the development approximators in the form of deep neural networks; cellular solids provide natural physical analog and-as as we show in this work-can can also be learned with automatic differentiation.
massively-paramen
composable function
To make progress towards the goal of learnable morphological computation, in this paper we com- bine metamaterials with deep neural networks into a framework we refer to as a neuromechanical autoencoder (NMA). While traditional mechanical metamaterials are designed for single tasks and actuations, here we propose designs that can solve problems drawn from a distribution over tasks, using a neural network to determine the appropriate actuations. The neural network "encoder" con- sumes representation of the task-in in this case, achieving particular deformation and nonlin- early transforms this into a actuations then displace the boundaries of the mechanical metamaterial inducing another nonlinear transformation due to the complex learned geometry of the pores; the resulting deformation corre- sponds the "decoder". By using differenniable simulator of cellular solids in an end-to-end way both the neural network parameters and the pore shapes so that they can work in tandem. The resulting system exhibits morphological computation in that learns split the processing task across the neural network and the physical mechanism.
linear actuations which play the role of the latent encoding. These
able to learn
The paper is structured follows. We first introduce the abstract setup for the neuromechanical autoencoder, followed by a brief description our mechanics model, geometry representation, and differentiable simulation. Although important for the success of method, the details of our dis-
2

---

Published as a conference paper at ICLR 2023
Patch
Patch 2
Patch
Patch
(a) Visualization of the decomposition of cell into patches. The pore shapes are parameterized by radii, ranging from 0.0 to 1.0. Radii all 0.0 represents completely closed shape.
(b) Visualization of sample initial geometry. The corners the cells are a geometic parameter; their constraints during NMA optimization are speci- fied by the dotted boxes.
Figure 2: Geometry representation of pores and corners.
cretization and solver for computational nonlinear elasticity problems are in the appendix. We then describe and detail results of our experiments, which include mechanical tasks, shape matching experiment, and new mechanical twist on MNIST classification. We end with related work and discussion on future steps.
METHODS
2.1
NRUROMECHANICAL AUTOENCODER SETUP
We describe the overall setup pictured in Figure 1 We begin by considering bounded do- main 'D C IR.2 (often square) on which our material exists (in this work we only consider the ac- tuation of 2D) geometries). When actuations described by displacement field u: 'D R2, where u(X) represents the displacement the par- ticle originally coordinate X D. For simplicity, assume u can be discretized and identified by miite-dimensional vector E RR The exact form of the discretization is based on a finite element method variant and is described in the appendix in Section A.1
applied on the material, its deformation can be
Next we specify distribution of tasks T and an associated loss function t4q:tt) RN RT + R, which depends on task descriptor ~ T, E R= and a displacement field specified by q. The loss function often only looks at the deformation of a subset of the material, such as the displacement of single point, but we are restricted this. The task descriptor is meant to generic: it can be a coordinate, an image, scalar parameter, etc.
To map task descriptors to displacements, we use neural encoder Eg R - R* and mechanical decoder D3 R + R The output of the encoder at l; is understood to be the latent dimension of the autoencoder, and represents the actuations to the mechanical structure. The goal is to choose parameters {8, o) minimize the loss over the distribution of tasks:
0" o" = arg min E (C(D) 0,0
Given V (DL( and VoL(Ds (t));: (F t) for ~ T, we can optimize the objective with standard first-order stochastic gradient methods. One difficulty is that D&(.) is implicit function of its inputs, computed by solving a partial differential equation (PDE). Furthermore, á represents geometric parameters defining the domain on which the PDE solved. To effectively compute derivatives of Dø,, we developed a JAX-based (Bradbury et al. 2018) differentiable elasticity simu- lator, as described in the next section.
2.2 DIFFERENTIABLE SIMULATION
We developed custom solver for static nonlinear elasticity problems which model the equilibrium of elastic materials under load. The goal is to have robust and end-to-end differentiable simulator for 2D neuromechanical autoencoders based on mechanical metamaterials. Given geometric design parameters, our solver simulates the structure described by the parameters and computes the gradient (adjoint) with respect to both geometric design parameters and boundary conditions (actuations).
3

---

Published as a conference paper at ICLR 2023
Figure 3: Visualization of the transla- tion task, and the unoptimized initial geometry. The goal is to translate the green pointer to anywhere within the blue square (such as the labeled dot on the right), using only linear actua- tions on the two sides. With the start- ing geometry. only horizontal transla- tions are possible.
Actuation
order to make the solver differentiable with respect to geometric parameters, we implement a ver- sion of isogeometric analysis (IGA) (Hughes et al.) 2005). finite element method (FEM) (Hughes) 2012) variant where both the underlying solution and geometry basis are based on B-splines. Us- ing B-spline patches allows to parameterize our geometry in flexible and yet robust way while maintaining a differentiable map from geometry parameters to PDE solution.
As our simulator is implemented entirely in JAX, backpropagate gradients directly through both the simulator and neural network using automatic differentiation and adjoint methods in tandem. In the sections, we describe the relevant physics and the geometric representation we used.
2.3 MECHANICAL MODEL
We give high level description of the mechanical model here, and detail further in the appendix. In the static equilibrium problems we consider, the solution is a displacement that minimizes some energy. The elastic properties are captured by a hyperelastic strain energy density function, which depends on the local deformation of the material and is independent of the path of deformation. For given deformation, the potential energy functional (u( the integral of the strain energy density over the material domain D. Given boundary conditions, the resulting physical deformation DD IR.2 is one that minimizes V/ (u) subject to boundary conditions:
u" arg min (u) uEH
where H is the set of all displacement fields that satisfy prescribed Dirichlet boundary conditions (expressed as equality constraints on the displacement field). To solve this in practice, we dis- cretize u and define standard representation of the geometry. NMA training is bi-level, where in the inner loop we perform the energy minimization using second-order methods. In the outer loop, the solution u* can be regarded as implicit function of the design parameters and boundary conditions, and gradients with respect to these can be computed using implicit differentiation.
2.4 GEOMETRY REPRESENTATION
The central unit of the metamaterials we design is the cell, a porous shape with a quadrilateral boundary. We initialize the geometry regular grid of square cells with simple square pore shapes, similar to that in Figure 2b) During training of the neuromechanical autoencoder, we modify this geometry minimize the expected loss over distribution of tasks.
To represent the geometry, we decompose the domain into B-spline patches, each with its own B-spline control points. Each cell is generally composed of four patches; we visualize the decompo- sition of a representative cell in Figure 2a The shape of the cell pore is defined by radii (illustrated by rz), whose values specify relative distance of the pore edge from the centroid of the cell (c.g.. cell having radii all 0.0 corresponds to completely closed cell). We combine all the radii in all cells into a radii array E [0, 1] which becomes one of our geometric parameters. For further flexibil- ity, we also allow the shapes of the cells to change within grid of cells. The corners of the cells, labeled e; in Figure 2bb are allowed to deviate within a specific box around its values in the initial square lattice-like geometry. Figure 2a shows cell that its corners perturbed during training. The deviation bound ensures that the shapes do not degenerate during NMA training. The array of corner locations and radii comprise our geometric parameters. The outer boundary of the structure is constrained not to change during NMA optimization, as this would otherwise create inconsistent boundary conditions between designs.
4

---

Published as a conference paper at ICLR 2023
Arti Cnntiiio Cationn 2
Actuation
Figure 4: After optimization, able to achieve both horizontal and vertical translation using linear actuations. The learned diagonal pore shapes allow compressive actuation to be translated into downwards motion, and tensile actuation into upwards motion. The neural network, jointly learned along with the geometry, successfully translates the goal coordinate to the appropriate actuations. Diagonal
Up
Down
40
Figure 5: Real-world design of the translation task. (a) the material in its starting unactuated con- figuration, (b-d) shows the structure in tension, compression, and one sided compression to achieve upwards, downwards, and diagonal pointer displacement. In (e-g) the reference picture (a) is over- layed on top the actuated material(b-d) to demonstrate the displacement the pointer.
2.5 DISCRETIZATION AND END-TO-ENT TN-DD DIFFERENTIABILITY
The details of our discretization are in the Appendix. We mention two important notes here. The first that careful selection geometric parameters is critical to being able to differentiate with respect them. In particular, given the geometric parameters to the B-spline control points representing the geometry of the model. The analogy in standard FEM would be that our "meshing" operation is fully differentiable. Part of the reason differentia- bility is always satisfied is that the cardinality and topology of the control points remain the same given any valid setting of geometric parameters. Another important note that all of our geometric parameters, fr, c), are only constrained by simple box constraints, so that first order constrained optimization with them straightforward. This parameterization robust the sense that for any value of the geometric parameters within the box constraints, we have a valid geometry.
construct differentiable map
3 EXPERIMENTS
3.1 TRANSLATION AND ROTATION
The first task we tackle is how to perform translation given a limited degree of control. Consider the setup in Figure 3 We have a x cellular solid fixed on the corners. The goal is to be able to move the green pointer in the middle of the solid to anywhere within the blue square given only horizontal displacements of the edges. The space tasks is a small box B, and the task is defined by a single coordinate; the task descriptor t IR? is sampled uniformly from B. The task deseriptor is mapped to two horizontal actuations by a simple fully-con fully-connected neural network (NN). The loss function is mean squared error of the green pointer after deformation from the goal.
5

---

Published as a conference paper at ICLR 2023
Figure 6: Single direction rotation with single actuation. The left figure de- scribes the setup and learned geome- try. Using single actuation applied equally on both sides of the metama- terial, we can achieve any single direc- tion rotation up to an angle /4 of the middle box.
Actuation
In the absence of dopain-specific design, the allowed actuations limit the achievable motions only left-right displacements. As demonstrated in Figure 3 (right), given a red goal point on the bottom right the best we can do with the unoptimized geometry is match the x-coordinate. If do end-to-end NMA training. however, we converge on the shape in Figure The learned geometry is able to achieve any translation within the blue square. The diagonal pore shapes enable translating compression of the material to downwards motion of the pointer, and tension to upwards motion. Here the joint learning of geometry and control offers a clear benefit: we converge nontrivial solution that discovers how to use its geometric nonlinearity with its NN controller.
To demonstrate transfer to the real world, we manufactured the resulting structure and qualitatively verified its behavior. As predicted by our simulation the neuromechanical autoencoder can facili- tate displacement a central point in the upwards via tension, downwards via compression, and diagonally via partial compression. See Figure for details.
The next task another mechanical task: rotation. We would like to see how the NN and metamate- rial can work together to learn translate linear actuation into rotation of part of the structure. The task description is now the angle of rotation: t E [-x, T.. We have two setups for this problem. In the first, use a small fully-connected NN map angle into single actuation, applied equally both sides of the metamaterial (Figure 6. We apply actuations on both sides to avoid translating the middle square in addition rotation. In the second, the NN maps into actuations, one applied left-right and one applied top-down (Figure 7). p. In both cases we consider a 7 x 7 metamaterial. The goal is rotation of the blue stick counter-clockwise by an angle & around the center; the loss function is mean square error from the goal of the two points on the ends of the stick.
In the first setup, we are able to achieve unidirectional rotation between [0. r/4 (Figure 6 right), while for the second setup, we can achieve bi-directional rotations between [-T/6, /6, x x/6] (Figure Without geometry and control co-design, we would not be able to achieve rotation with linear actu- ation without an intuition-driven design, but the joint NMA training is able to make good progress.
Accuatica
Figure 7: With two inde- pendent actuations, we can achieve bi-directional ro- tation from [-T/6, /6). One actuation is applied equally on top and bot- tom, and the other applied equally left to right.
Artuancom
3.2 SHAPE MATCHING
Next we consider a much digher-dimensional task space. Given a family of shapes parameterized by 2-dimensional coordinates, we would like to design a mechanical decoder and neural encoder that can map coordinates actuations deforming the structure resemble sampled shape from the family as closely as possible.
In particular, consider family generated by log Gaussian process in polar coordinates, ap- proximated via random Fourier features (Rahimi & Recht 2007). Given a metamaterial with large central pore, such in Figure Sal we would like to deform it to match any of the shapes in the

---

Published as a conference paper at ICLR 2023
Only NN optimized
Geometry optimized
(a) Undeformed configuration of both materials.
(h) Loss curve
Figure 8: Pore shape matching experiments. (a) The undeformed configuration of the unoptimized geometry and the optimized geometry. The 12 actuations applied the material are depicted with the square brackets. (b) The stochastic loss during training. The optimized geometry achieves a significantly smaller loss since it is able to capture the finer features in the shapes. The geometry can tune itself to the particular random shapes that comprise the dataset.
Brgal
Toget
Figure 9: Comparison on random shapes. The optimized geometry able to capture finer features of the dataset compared to the non-optimized geometry. Note that our loss function is rotation- invariant, so the orientation of the shapes do not necessarily match the orientation of the pores.
family. The task description is an dimensional array of coordinates defining the shape. A fully-connected neural network translates these to 12 actuations applied around the material (as shown in Figure Bap. The final loss function is an loss between the control points defining middle pore after deformation and the points defining the shape. When comparing, normalize the scale of both shapes, and perform Procrustes analysis for rotation invariance.
We train one version where the geometry and neural network are optimized jointly, and one version where only the neural network is optimized for the starting geometry. Figure Sa shows the non- optimized and optimized geometry. After learning, Figure 9 shows qualitative results of how well the jointly learned metamaterial compares with the control-onle material. Jointly learning geometry for the shape family allows capture much finer features in the target shape. In Figure 8b

---

Published as a conference paper at ICLR 2023
(a) The starting color map. (b) The optimized color map.
(c) Color map with slits.
Figure 10: The shapes for digital MNIST, along with labeled locations for actuations. The learned color map is highly nontrivial, and allows to perform underactuated control of the seven slits. Note that the undeformed configuration happens look like the digit "5", which surprising as the average of digits across the seven-segment display is not 5.
visualize the (stochastic) loss during training. The jointly learned metamaterial converges significantly lower loss value, showing the benefit of harnessing the geometric nonlinearity.
3.3
DIGITAL MNIST
For our last task we attempt digits. Towards this we add an additional design variable for the material: color. Our starting metamaterial is pictured in Figure T0a version of metamaterial that is originally assigned color value O everywhere. We treat color, parameterized by a B-spline patch over the metamaterial, as an additional geometric design parameter that can be optimized with NMA training.
a mechanical seven-segment display for classifying MNIST
2
NN
7
NN
Actuations
Referescee Coufiguration
Deformed Configuration
The setup for the digital MNIST task. The MNIST digit is fed into neural network, which then produces actuations to deform the material in six locations. After deformation the slits show digital eeven-segment representation of the MNIST digit.
Figure 11:
Our input to the neural network is an image sampled from the MNIST dataset. The neural network then produces actuations that deform the metamaterial to produce seven-segment representation of the MNIST digit when viewed through small slits. Figure 10b visualizes the learned colop map, and Figure 10c shows the structure with slits added. The loss function is manually specified for each digit, e.g. an MNIST digit has a label of "1" then the right two slits should contain color value 1.0, while the rest should contain 0.0. The full setup is displayed in Figure IT] with samples after training displayed in Figure [2 Although this can be learned from scratch end-to-end, to speed up training first learned colors and actuations be able to reproduce all 10 digits, and then trained
8

---

Published as a conference paper at ICLR 2023
small feed-forward neural network to match the actuations for each digit. We then set up the entire pipeline and finetumed end-to-end for better performance. We note that the segments are controlled by only 6 actuations, so by restricting the family of objects displayed to the 10 digits, we allow the neural encoder and mechanical decoder to learn underactuated control of all segments. Additional samples are presented in the Appendix. We also note that the pore shapes did not have to change significantly to accomplish this task. The only "geometry" design was through the coloring, which as visualized in Figure I0b turns out be highly nontrivial.
4 DISCUSSION AND RELATED WORK
Differentiable Simulation The abundance of differentiable simulators has demonstrated their usefulness in designing novel systems. Hu et al. (2019) developed a differentiable simulator with hand-written custom CUDA physics kemelss that enabled material inference, control a soft walker, and co-design of swinging robot arm. Sanchez-Gonzan (2020) presents an ML framework to model a variety of phys- ical domains solve forward and inverse problems using graph neural network approach. |Mozafaar & Cao (202T) de- veloped a differentiable finite element simulator to control and infer material parameters within the context of additive manu- facturing processes. Liang et al. (2019) developed differen- tiable cloth simulator, and Ham et al. (2119) automated the cal- culation of weak shape derivatives within the context of finite elements to solve PDE constrained shape optimization prob- lems. our work, our differentiable simulator is developed specifically to aid in neuromechanical autoencoder design.
et al.
Mechanical Metamaterials As the rational design of nonlin- mechanical materials is often unintuitive, modern machine learning approaches have enabled faster design. Deng et al. (2022) coupled neural accelerated mass spring model that fa- cilitated an evolutionary approach to design functional struc- tures. Mao et al. (2020) applied generative adversarial net- works to design unit cells for architected metamaterials, Ku- mar al. 2020) introduces a novel class of anisotropic meta-
9
Figure 12: Some MNIST samples and results. Additional samples available in appendix.
materials and a machine learning method for the inverse design of their geometry given desired elasticity properties, |Battson et al. (2020) learned reduced order model to speed up simulation of cellular metamaterials, Xue et al. (2020) introduced homogenization approach for cellular meta-
materials, and Xue & Mao (2022) introduced mapped shape approach to design metamaterials to fit prescribed strain energy curve. Our work uses classical gradient/adjoint methods to opti- mize the geometric parameters, but could be combined with machine learning methods to speed up simulation and hence faster NMA training.
4.1 LIMITATIONS AND FUTURE WORK
We introduce the framework of neuromechanical autoencoders, inspired by the biological co- evolution of control and morphology. We present method for automatic design of these systems, and show a number of results that produce nontrivial behavior through co-design, both in simu- lation and in real-world. We believe this is small but significant step in the road designing cethanically-itteligeee systems. The two major bottlenecks in our approach is the runtime of PDE solving and geometry parameterization. Fast PDE solving, especially for nonlinear PDEs such as the ones we use, is a very active area of research, and is crucial to scaling up NMA design. In terms of geometry parameterization, the key is to find a space of materials that have a complex range of mechanical deformation properties and yet are easy to simulate. For this paper, 2D cellular solids with nonuniform pore shapes were a great ansatz, but a future work could understand and quantify how much "computation" these materials can do. Scaling up to 3-dimensional intelligent mechanical models, as well as including dynamics, would significantly improve the computation capabilities, but would require much faster solvers. This is a key focus in our further work.

---

Published as a conference paper at ICLR 2023
4.2 ACKNOWLEDGEMENTS
We would like to thank Alex Beatson, Geoffrey Roeder, Jordan Ash, and Tianju Xue for early conversations around this work. We also thank PT Brun for assisting with fabrication. This work was partially supported by NSF grants IIS 1002207777 and OAC-2118201, the NSF under grant number 2127309 to the Computing Research Association for the CIFellows 2021 Project, and a Siemens Phl) fellowship.
REFERENCES
David N Beal, Franz S Hover, Michael Triantaffllou, James Liao, and George Lauder. Passive propulsion in vortex wakes. Journal of Fluid Mechanics, 549:385-400 2006.
Alex Beatson, Jordan T Ash, Geoffrey Roeder, Tiajju Xue, and Ryan P Adams. Learning com- posable energy surrogates for PDE order reduction. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 338-348, 2020.
Katia Bertoldi, Pedro M. Reis, Stephen Willshaw, and Tom Mullin. Negative Poisson's ratio behav- ior induced by an elastic instability. Advanced Materials, 22(3):361-366, 2010. ISSN 09359648.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne JAX: composable transformations of python+numpy programs. 2018. http::
www./ooooooooooooooooooooooooooo j/jax
Steve Collins, Andy Ruina, Russ Tedrake, and Martijn Wisse. Efficient bipedal robots based on passive-dynamic walkers. Science, 307(5712) 2005.
Bolei Deng, Ahmad Zareei, Xiaoxiao Ding, and James Weaver. Inverse design of mechanical metamaterials with target nonlinear response via neural accelerated evolution strategy. Advanced Materials.. 2022. ISSN 0935-9648.
Jeff D Eldredge and David Pisani. Passive locomotion of a simple articulated fish-like system in the wake of an obstacle. Journal of Fluid Mechanics, 607:279-2899 2008
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 12(1):5721, October 2021. ISSN URL
2041-1723. doi: 10. 241467-021-2588
100 1038/
David A. Ham, Lawrence Mitchell, Alberto Paganini, and Florian Wechsung. Automated shape differentiation in the unified form language. Structural and Multidisciplinary Optimization, 60 (5):1833-1820, 2019.
Helmut Hauser, Auke J Ijspeent, Rudolf M Füchslin, Rolf Pfeifer, and Wolfgang Maass. Towards theoretical foundation for morphological computation with compliant bodies. Biological Cyber- netics, 105(5-6):355-3700 2011.
Fabian Heinemann, Steffen Puhlmann, Clemens Eppner, José Élvarez-Ruiz. Marianne Maertens, and Oliver Brock. A taxonomy of human grasping behavior suitable for transfer to robotic hands. In 2015 IEEE International Conference on Robotics and Automation, pp. 4286 4291-4291. IEEE, 2015.
Philip Holmes, Robert Full, Dan Koditschek, and John Guckenheimer. The dynamics of legged locomotion: Models, analyses, and challenges. Society for Industrial and Applied Mathemutics Review, 48(2):007-344, 2006.
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Fredo Durand. DiffTaichi: Differentiable programming for physical simulation. In International Conference Learning Representations, 2019.
Thomas J.R. Hughes. The Finite Element Method: Linear Statie and Dynamic Finite Element Anal- ysis. Courier Corporation, 2012.
10

---

Published as a conference paper at ICLR 2023
Thomas J.R. Hughes, Austin Cottrell, and Yuri Bazilevs. Isogeometric analysis: Cad, finite ele- ments, nurbs, exact geometry and mesh refinement. Computer Methods in Applied Mechanics and Engineering. 2005. ISSN 0045-7825.
Mare Jeannerod. Grasping, historical perspective. Sensorimotor Control of Grasping: Physiology and Pathophysiology, pp. 127, 2009.
Siddhant Kumar, S. Tan, Li Zheng, and Dennis M. Kochmann. Inverse-designed spinodoid meta- materials. Npj Computational Materials, 6(1), 2020.
George V Lauder, Erik J Anderson, James Tangorra, and Peter GA Madden. Fish biorobotics: kinematics and hydrodynamics of self ceff-propulsion Journal of Experimental Biology, 210(16): 2767-2780, 2007.
Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yunwei Mao, Qi He, and Xuanhe Zhao. Designing complex architectured materials with generative adversarial networks. Science Advances, 6(17):419-4419, 2020.
Mojtaba Mozaffar and Jian Cao. Additive manufacturing process design with differentiable simula- 220.11991999
tions. CoRR, abs/2107.10919 2021. URL
R. Ogden. Non-linear Elastic Deformations. Dover Civil and Mechanical Engineering, Dover Publications, 1997.
Chandana Paul. Morphological computation: A basis for the analysis of morphology and control requirements. Robotics and Autonomous Systems, 54(8):619-630 2006.
Les Piegl and Wayne Tiller. The NURBS Book (2nd Ed.). Springer-Verlags Berlin, Heidelberg, 1997. ISBN 3540615458
M. J. D. Powell and Ph. L. Toint. On the estimation of sparse hessian matrices. SIAM Journal on Numerical Analysis, 16(6):1000-100 1979.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Youcef Saad and Martin H. Schultz. Gmres: generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 7(3):866~ 869, 1986.
Alvaro Lanchez-Gonzaler Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8459-8466 PMLR, 13-18 Jul 2020.
Karl Sims. Evolving virtual creatures. Proceedings of the 2 Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '94, pp. 15-22, New York, NY, USA, 1994. Association for Computing Machinery. ISBN 0897916670, doi: 10.1145/192161.1192117 URL
hhttps://doi. wwt.:ooggll00....
http://666..00/11011111111221222 http://....155192210 1921677 92]
Lena H Ting and Lucas McKay. Neuromechanies of muscle synergies for posture and movement.
Current Opinion in Neurobiology, 17(6):622-228, 2007.
Tianju Xue and Sheng Mao. Mapped shape optimization method for the rational design of cellular mechanical metamaterials under large deformation. International Journal for Numerical Methods 2022.
in Engineering, 123(
Tianju Xue, Alex Beatson, Maurizio Chiaramonte, Geoffrey Roeder, Jordan T. Ash, Yigit Mengue, Sigrid Adriaenssens, Ryan P. Adams, and Sheng Mao. A data-driven computational scheme for the nonlinear mechanical properties of cellular mechanical metamaterials under large deforma- tion. Soft Matter, 16(32):7524-534 2020.
11

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 DISCRETIZATION AND IGA
Parent Space
Patch 3
74
72
7
Patch 2
75
Patch 4
y
Patch
Figure 13: Visualization the decomposition of cell into patches, Each patch pulled back into a parent space, which is a B-spline knot span. Quadrature and integration happens in this space. The pore shapes are parameterized by radii.
We discretize the geometric domain into P isogeometric patches. All patches use the same B- spline basis functions ( 11) Piegl Tiller 1997) with i, E [mp] and E, E [0, 1], where Mp is the number of control points. The domain of and v correspond to the parent domain of each patch (Figure 13). In the B-spline literature, the parent domain often referred to as the knot span. The basis functions are piecewise polynomial with degree specified as parameter. In all of our experiments, we use piecewise quadratic B-spline functions. Each B-spline basis function corresponds
degrees of freedom in 2D) space, i.e., each patch p E [P] has Up X ny x degrees of freedom (xz) and y²").
control point, and each control point represents
The mapping from the parent domain of each patch the physical domain given by a linear combination of B-spline basis functions, where the weights of the linear combination are given by the control point coordinates. Explicitly, the mapping function op from parent space of patch p to physical space is given by
np
(P
B (E,
where [x . are the control points parameterizing the mapping. In our simulator, represent the reference configuration with reference control points [XY, Yý"]: these are determined by our geom- etry parameters (r, c} through differentiable map. We then parameterize the deformed geometry with the same basis, using deformed control points [a>)] y²]. For given deformation, the integral represetiig the potential energy can be computed by pullback in the parent domain, using standard Gaussian quadrature as is standard in FEM (Hughes et al. 2005).
Dirichlet boundary conditions of the type we use in this paper can be represented as constraints on subset of the control points. For each boundary condition, the corresponding reference and deformed control points are prescribed to have particular displacement values. Furthermore, since our geometry is decomposed into multiple neighboring patches, the control points must also have incidence constraints amongst them. These are kept track of using constraint groups, where each group has representative element.
A.2 MECHANICAL MODEL.
In the static equilibrium problems we consider, the solution a displacement that minimizes some energy function. In particular, nearly incompressible Neo-Hookean material model (Ogden
12

---

Published as a conference paper at ICLR 2023
1997). in which the elastic properties captured by hyperelastic strain energy density function. This function, (F), W I= - I. is independent of the path of deformation and a function of the deformation gradient tensor, Fij 3u,/0x + where X D C R² represents the position of a particle in the undeformed reference configuration, and u (X) is the displacement field. Here,
# 2 (41 - 2 log J) + J)²
(FF)
where / (F) 41
tr F). and μ E/2(1 and E/3(1 2v) are shear and bulk moduli of material with Young's modulus F. and Poisson's 's ratio respectively. This is a standard choice for hyperelastic material modeling that transfers well to the real-world. We can solve for the displacement by finding the stationary point of the potential energy functional (u),
du
"t" = arg min (u) WEH
(u( = W(F)dX= W
+1
dX
(3)
XX X-X
OX
where H is the set of all displacement fields that satisfy prescribed Dirichler boundary conditions (expressed as equality constraints on the displacement field). To solve this in practice, we dis- cretize u and define standard representation of the geometry. Abstractly, the solution u" can be regarded as an implicit function of the design parameters and boundary conditions. The solution u² can be computed in discretized form using standard second-order optimization algorithms, and gradients can be computed using implicit differentiation.
A.3 END-TO-END DIFFRENTIABILITY
We would like reiterate that our differentiability conditions are satisfied, so that we can train neuromechanical autoencoders end-to-end. We first define a differentiable map from geometry pa- rameters and Dirichlet boundary condition values into the reference B-spline control points. This then used to construct the IT, HI functions, both of which are differentiable. The global vector then passed into a black-box optimizer to produce the solution q², Gradients with respect to the solution of the optimizer are computed using adjoint optimization. The solution q" is then mapped back into local coordinates using HT, and is used to compute the loss function L.
This pipeline ensures that have a differentiable map from geometry parameters and boundary conditions (actuations) to the NMA task loss function, so we can proceed to train the NMA objective using stochastic gradient descent. In the section, demonstrate specifie applications.
A.4 SOLVER DETAILS
After discretization to 9. we solve the energy minimization using Newton's 's method with incremental loading. The Hessian of the energy is assembled in sparse form using the trick from Powell & Toint (1979). Using the discretization of the system we automatically derive the sparsity pattern of the Hessian, and then construct appropriate binary vectors to perform Hessian-vector products with. We then reshape these into CSR matrix representation of the Hessian.
The sparse linear systems are then solved by GMRES (Saad Schultz 1986) preconditioned by an incomplete LU decomposition. Since the energy involves log det F, where F' is the deformation gradient, taking a finite step can lead numerical blowup. Therefore our incremental loading is adaptive, and a line search is performed to avoid inversion of elements in the geometry.
A.5 EXPERIMENTAL DETAILS
All B-spline patches used were quadratic and contained 5 control points. Quadrature was done by degree 5 Gauss-Legendre Most computation was done on NVIDIA RTX 2080 GPUs. Each solve instance was done on a single GPU, and mini-batching was done by parallelizing with MPI. Each MPI task used single GPU. The radii parameters were clipped to [0.1, 0.9) to aid solver stability.
A.5.1 TRANSLATION TASK
The learning rate was 0.0001 * M where M is the number of MPI tasks. In this case, we used 8 MPI tasks. The neural network was fully-connected network with activation sizes: 2 2 30 3000300 30 10 - (including input/output). The final layer was clipped by a tanh and multiplied by a maximum displacement of 60% of cell width.
13

---

Published as a conference paper at ICLR 2023
A.5.2 ROTATION TASK
The learning rate was 0.001 M for single and 0.01 M for double rotation. M is the number of MPI tasks. In this case, we used 8 MPI tasks. The neural network was a fully-connected network with activation sizes: - 20 actuation. The final layer was clipped by tanh and multiplied by maximum displacement of 60% of cell width.
1 (including input/output) or 1 10 for the double
A.5.3 SHAPE MATCHING TASK
The learning rate was 0.0001 M where M is the number of MPI tasks. In this case, we used 16 MPI tasks. The neural network was fully-connected network with activation sizes: 98 200 200 12 (including input/output). The final layer was clipped by a tanh and multiplied by maximum displacement of 60% of cell width.
A.6 DIGITAL MNIST TASK
Initially we learned colors and actuations for a lookup table of 10 digits. This was trained with learning rate of 0.01 M where M is the number of MPI tasks. We used 10 MPI tasks, one per digit. We clipped the maximum displacement to 60% cell width using tanh. Afterwards, we trained fully-connected neural network to map MNIST digits to the actuations of the corresponding digit. We then put the neural network to map directly to actuations, and finetuned end-to-end with learning rate of 0.0001.
14

---

Published as a conference paper at ICLR 2023
A.7
ADDITIONAL PORE MATCHING RESULTS
MI Coundry
Nt only
Deger
Coomedyy
Uaper
Bage:
BM
Kk Oonedry
seby
15

---

Published as a conference paper at ICLR 2023
A.8
ADDITIONAL DIGITAL MNIST RESULTS
5
3
4
16


Reviews:
[{"review_id": "U8iKCTKI-5", "reviewer": "Reviewer_S3up", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce neuromechanical autoencoders (NMA) -- neural encoder combined with a differentiable mechanical decoder -- to learn to perform morphological computation using gradient descent. The authors show how it is feasible for the decoder to perform translation, rotation and matching shapes specified to the encoder via actuations output by the encoder. They also demonstrate the performance of NMA on a material they manufactured to verify how the model works in the real-world. ", "summary_of_the_review": "I find the presented work to be highly novel and potentially impactful for the intersection of machine learning and multiple basic sciences. I believe this line of work is important and it looks like the domain is very new and understudied. I find this work to be very relevant to representation learning and the ICLR audience, and hence recommend acceptance. That said, I am not an expert in this area and am willing to read other reviews and update my score based on potential weaknesses I may have missed to identify.", "strength_and_weaknesses": "Strengths:\n- The paper presents a highly novel and technically sound integration of neural network-based encoders with differentiable mechanical decoders. This combination potentially has several applications in the real-world such as robotics, material design, 3D printing, etc. \n- The digital MNIST task that the authors develop -- in which the neural encoder receives an RGB image of a digit as input and the mechanical decoder produces deformations to the metamaterial that classifies the digit -- is very intriguing and a great demonstration of learning (with gradient descent) a solution to the common and accessible MNIST task through morphological computation.\n- The authors share their code as part of the submission, this should is helpful in more thoroughly understanding the proposed work.\n\nWeaknesses:\n- I found it hard to grasp Methods subsections 2.3 and 2.5. I am not an expert in this domain and I'm not surprised that I find it hard to read these sections, but it would be great if the authors could please see if they can further improve readability of these sections for the broader audience.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "F8PDdNyS--2", "reviewer": "Reviewer_Fw7y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "I thank the authors for their rebuttal replies and for further improving the quality of this paper. I think my score is still accurate and I look forward to seeing this work published.\n\nThe paper proposes a differentiable model for learning meta material layouts and neural network encoder that can produce actuations to control these materials. They show through a number of simulations that this works with simple translations, rotations and complex shapes. They also build this material and show that it works. Finally, they show how this combined framework can be trained to translate MNIST digits into 7 slits representations.", "summary_of_the_review": "The authors propose a model combining a neural network controller and a learned geometry. This is a very creative solution to a problem in material sciences (I suppose) and it is implemented with some cool technical methods based on spline representations, adjoins computation and extended efforts to build a differentiable system that works under a number of physical and geometric constraint. I think that this is a really project and would be happy to see it published – supposing that other reviewers which are more familiar with this research area agree on its novelty and soundness.", "strength_and_weaknesses": "Strength: The paper is very well written, easy to follow and fun to read. Even for a reader unfamiliar with meta materials and the associated physics, they manage to write an engaging scientific exploration. The proposed model is very creative and works at an exciting intersection of material and data science.\n\nWeaknesses: It would have been great to include a tiny bit more detail about technical things, for instance, how is the adjoins method implemented, which PDE solver is used etc. However, I do realise that this might come at the expense of making the paper less accessible.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7Z_0nlOOqy", "reviewer": "Reviewer_twBy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This works presents a co-design of metamaterials based mechanical/morphological actuation and neural network-based control. The control tasks are allowed to be image, coordinate on scalar parameter based on the end goal, which are mapped on to a latent dimension (using an encoder) matching the number of linear actuations. The decoder is a differentiable simulation that solves for equilibrium of non-linear elastic materials under mechanical load that takes the geometric design parameters and actuations (output from encoder). The solver is made differentiable wrt the geometry parameters (represented as a porous shape with quadrilateral boundary) by adopting isogeometic/FEM analysis and B-spline basis to solve the PDE. The results show that the geometry and control co-design is able to achieve translation, rotation and shape matching efficiently and better than geometry-only scenario.  It is interesting to see the digital MNIST benchmark and the application of this approach to that case", "summary_of_the_review": "The geometry and control co-design using differentiable simulator and neural network combination shows good results, but no algorithmic innovations in machine learning are proposed in this work. More details need to be provided about the approach to better assess its applicability for other general scenarios.", "strength_and_weaknesses": "Strengths:\n\n* The use of neural networks to converts tasks to actuations and that used along with the simulation-based decoder in a single differentiable framework shows great promise in achieving nontrivial displacements\n* The adoption of cellular solids and B-spline patches into the mechanical simulation/decoder seems to work well to produce stable solids.\n\nWeaknesses:\n* No new algorithmic/machine learning approaches have been proposed\n* The work is just coupling the a simple multi-layer perceptron to a well-designed simulator (PDE solver) to achieve the joint optimization of geometry and boundary conditions\n* Time complexity of training/learning in this approach is not discussed\n* What is the motivation for using L2 loss in one case and L1 in the other?\n* How expensive is the generation of training data; is this approach scalable to train with larger datasets for more complex deformations?\n* What is the reason to choose the Neo-Hooken material model?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5-NH34UTcya", "reviewer": "Reviewer_fAWH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "= Update after rebuttal =\n\nI thank the authors for their response to my comments. I maintain my recommendation for acceptance.\n\n= Original review =\n\nThe authors show how to optimize the shape of an elastic material, as well as neural networks to control its actuators, imn order  to make the material perform various mechanical tasks. \n\nBoth the material's  shape and the controlling neural network  are optimized by gradient descent for various task domains. Given a target action (which,  depending on the  domain, might be a location for  a point, a shape,  or even an MNIST digit image to be classified), the neural network translates a specification of  the action into adequate actuations, through which the optimized shape performs the  specified action.\n\nThe authors apply their method to translation/rotation of a point on the material, controlled deformation of a  shape, and -  remarkably - translating MNIST images into seven-segment digit displays through deformation of a colored shape  seen through slits.\n\nThe authors also show a working real-life implementation of the system, demonstrating practical applicability.", "summary_of_the_review": "It's new, interesting, and seems to work well. Easy accept.", "strength_and_weaknesses": "Strength:\n\n- The method is novel (to my knowledge) and interesting. While there has been much work on optimizing morphologies and controllers jointly (e.g. Karl Sims and other \"virtual creatures\" models - these should be cited!), as well as optimizing elastic shapes (e.g.  Sodarace), the system described here  looks like nothing I'm aware of.\n\nWeaknesses: \n\n- The only minor defect I can see is that some points in the explanation could be clarified - see below", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%