Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation
Authors: Jin Woo Lee
Jaehyun Park
Min Jun Choi
Kyogu Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Differentiable Audio Signal Processing
Physical Modeling
Musical Sound Synthesis
Physical Simulation
Primary area: speech_and_audio
TL;DR: We propose a model that simulates a musical string instrument from the physical properties.

Abstract: While significant advancements have been made in music generation and differentiable sound synthesis within machine learning and computer audition, the simulation of instrument vibration guided by physical laws has been underexplored. To address this gap, we introduce a novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework. Our model leverages mechanical properties and fundamental frequencies as inputs, outputting string states across time and space that solve the partial differential equation characterizing the nonlinear string. Empirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared to existing baseline architectures. The code and demo are available online.

Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation
Jin Woo Lee Seoul National University j jinwlee@snu. ac.
Jaehyun Park Seoul National University lotussoh@snu. ac.kr
Min Jun Choi
Kyogu Lee +it
Scoul National University choimj21@smu aw..nnac..... ac. kr
Scoul National University kgleeSsnu. ac kr
Abstract
While significant advancements have been made in music generation and dif- ferentiable sound synthesis within machine learning and computer audition, the simulation instrument vibration guided by physical laws has been underexplored. To address this gap, introduce novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within neural network framework. Our model leverages physical properties and fundamental frequencies as inputs, outputting string states across time and space that solve the partial differential equation characterizing the nonlinear string. Em- pirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared existing baseline architectures. The code and demo are available online.
Introduction
The investigation of wave propagation along strings, encompassing both theoretical and experimental dimensions, has persisted for well over a century [1] In the relentless pursuit verisimilitude and expressive fidelity in simulating wave phenomena, numerous studies have been investigated to bridge the gap between theoretical underpinnings and empirical sound measurements [3]. Advancements leveraging computational power to mimic the intricate physical processes in musical instruments have given rise to numerical sound synthesis, now a cornerstone in the field virtual sound synthesis 4 5) Schwarz (6] presents systematic study of parametric and physical models for music audio synthesis. Parametric models include signal models, such as spectral modeling synthesis 77 Physical models encompass techniques such modal synthesis, digital waveguides time-domain (FDTD) methods 4 10].
9. or finite-difference
Over recent years, the advancement of hardware acceleration for artificial intelligence has enabled the emergence of numerous techniques for neural audio synthesis (TT) including autoregressive generation [12] adversarial training [13] with phase coherence [14], and approximated physical models 15 The concept of differentiable digital signal processing (DDSP) was first introduced by Engel al. [16], aiming to incorporate known signal model into neural networks to achieve a domain-approns appropriate inductive bias. While the DDSP model can be considered a differentiable
MMsic and Audio Research Group (MARG), Department Intelligence and Information Artificial Intelligence Institute of Scoul National University (ATIS)
Interdisciplinary Program in Artificial Intelligence (IPAI)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Motion Synthesis
Sound Synthesis
Mechanical Properties
Control Parameters
Differentiable Modal Synthesis for Physical modeling
Intial conditio,
x
Tansion.
Süffress,
Lossiness,
Pithh Time, Pickup position...
System overview. The DMSP model encodes the physical properties of a string (e.g.. tension, stiffness, damping, and initial conditions) to estimate the displacement of the string plucked pitch fu at given time t [0, oo) and position x E Q By concatenating the DMSP outputs over the domain (æ, t) E f2 x [0, oo), the simulated motion of the string can be visualized. Reading the outputs at particular position æ allows hearing the synthesized string sound, akin to listening with a stethoscope the pickup position.
Figure 1:
version of spectral modeling synthesis, a wide variety of works have explored the differentiable implementation of other audio signal processing methods, such as the subtractive method [I7]. waveshaping [18], and frequency modulation [9 [9] Subsequent research has demonstrated various applications of DDSP, including music performance synthesis
[2T). speech synthesis and voice conversion 22 23 and sound effect generation (24). Renault et al. [25] extended DDSP to create a
polyphonic synthesizer, explicitly modeling properties specific to piano strings, such as inharmonicity and detuning induced by string stiffiness, based on a parametric model of these phenomena [26). This model efficiently synthesizes piano sound from MIDI input, achieving high mean opinion score on naturalness. However, it still shows room for improvement compared to sampling-basee based methods 27 and physical modeling methods (28).
Despite the growing recognition of DDSP as promising sound synthesis methodology, its extension physical modeling remains underexplored. Schlecht al. (29 have presented physical modeling using Fourier Neural Operator (FNO). They train recurrent-type FNOs to learn state transitions from data spanning a few initial timesteps in the simulation and then test generalization to the subsequent long-range data. Although they have demonstrated encouraging outcomes, their approach has room for improvement in that it is unconditional, meaning that it is challenging to generalize over dynamic scenarios (e.g., glissando, vibrato). In the context of rigid-body contact sound synthesis (30 few studies investigated the efficacy of training neural networks supervised by finite-element method propose neural network that predicts contact sounds from voxelized objects, inspired by the modal technique that synthesizes sound using eigenvalues and eigenvectors. Diaz al. (35) leverage differentiable infinite impulse response filter to synthesize contact sounds from rasterized occupancy grids in an end-to-end manner. These methods can interactively synthesize sounds for various contact conditions and materials with notable efficiency, circumventing the need for an offline optimization process typical of modal techniques. However, these methods, which resort to the FEM solver, vulnerable in modeling the dynamic behavior or in simulating the motion of the object.
(FEM) solvers. Jin et al. [33 34
In this regard, we propose a novel model for simulating the spatio-temporal motion of nonlinear strings, integrating modal synthesis and spectral modeling within a neural network framework. The proposed model leverages physical properties and fundamental frequencies inputs, outputting string states across time and space that solve the partial differential equation (PDE) characterizing the nonlinear string. Empirical evaluations demonstrate that the proposed architecture achieves superior accuracy in string motion simulation compared to the baseline architectures. The main contributions follows:
We present differentiable modal synthesis for physical modeling (DMSP) that simulates dynamic nonlinear string motion by synthesizing sound using the physical properties of the string.
To the best of our knowledge, this is the first differentiable approach that can synthesize the motion and the sound of musical strings with a dynamic control over the pitch and the material properties.
We provide an extensive empirical evaluation demonstrating the importance of modal decomposi- tion and the proper choice of loss function.

---

2 Background
2.1 Physical Modeling of Musical String Instrument
Linear Damped Stiff' String, The string model discussed in this paper is damped nonlinear stiff string. To introduce the nonlinear string, we first formulate the governing equations for the damped linear stiff string system and derive the corresponding modal solution.
On = 1~0m0 x² Qrzzzz - 20,0,u
(I)
The linear string of its length L, vibrating with wave speed y, stiffness is and frequency-indeeeeddntt damping factor on, is described by [qquation Given the initial conditions (IC) for z Ω L/2, +L/2] as u(z,0) 0) un (a) and 0, 0) = O, and appropriate boundary conditions (BC), the corresponding solution u(x, f) represents the motion of damped linear stiff string. Particularly, for a clamped boundary condition, i.e., 2(=L/2,t) t) = O, ((22,t/ = 0 for all f E [0), a modal solution can be obtained as follows.
u(x, = Σ (x(x)T((()
(2a)
n=1
sin Pm I/2 sinh vn + 02 cos
cos L,22 H4 (00s)10
(x) = c1 sin
cosh 2
(2b)
sinh Vn L
cosh U, L/2
(t) = e -aat cos 11 162 + 22, o
(2c)
The derivation of [quaatio 2 can be found in Appendix A The allowed values of pen and Vn. determined by the boundary conditions, while the coefficients ej and 02 are determined using the initial condition En=1 (x) (z)). Determining these values typically requires an offline numerical solving process, where the obtained values can be stored in memory for real-time computation of u(xxt) The stiffness modeled by the 4th order derivative induces a hyperbolic solution, resulting in non-integer multiples of the mode frequencies Wn, which leads to physical inharmonicity, The damping factor dg causes exponential decay in the temporal amplitude.
Aufar
(iz)
Nonlinear Damped Stiff String. The generalization of the linear wave Equation to nonlinear string vibrations is first introduced by Kirchhoff [36] and Carrier (37]. The Frrchhoff-Carrie system models elastic strings in two dimensions, and when extended so that transverse and longitudinal motions are coupled, phantom partials can be exhibited, resulting richer timbre 38 A model of such planar string vibration is as follows [39 40)
Figure 2: The planar string system.
Ouu -20rzu 72
O2 (g" + +pp) - 0120 u 2000,u + 20, 2,,0,
(3a)
72 a² O2 (q²) 2000s5 20 0,00m5
0u5 22020xx5
(3b)
Here, u(ar 1) and ((x, 1) represent the transverse and longitudinal displacements of a string, respec- tively, for all (xr, 1) Ω (0, o0). q Ōzu u and p = a. serves as the auxiliary coupling system, as 0:q = 0_0,u and 0,p = 020, 0, 41 A more detailed background on the derivation of Equattin 3 can be found in Appendix B] The boundary condition, which may vary depending the string being modeled, is chosen to be that of the clamped boundary condition:
u( t) = 0, u(x t) = O, v(z,t) E 02 x (0,
(4)
Given an initial condition Ug := u(x 0) defined on æ Ω2 the solution u(x, t) associated with the condition of Eqquatio 4immulases the motion of the string for physical modeling and sound synthesis of string instruments. Due to the coupling between Equation 3a and [Equatio 3b) the obtained solution exhibits features found in elastic strings, such as pitch glide and phantom partials. These features become more pronounced for larger displacements and are difficult to approach separately as in the linear case. The solution can be approximated through various physical modeling techniques such as finite-difference time-domain 42]. digital waveguides or functional transformation
method [4]
3

---

Table Comparison between methods. Computational complexity refers to the inference scenario.
System taxonomies
Computational complexity
Method
Physical
Nonlinear
Differentiable
Pre-computation Synthesis
Modal
x
X
(NN))
(1)
FDTD
/
X
N/A
O(2,N))
DDSP
x
/
J
N/A
((N))
DMSP-Hybrid
/
/
V
(N
O(1)
DMSP
V
/
/
N/A
O(1)
Finite-difference Time-domain. One straightforward approach to tackling nonlinear PDEs such as Equation would be employing finite difference approximation. This method, commonly known as finite-difference time-domain (FDTD), has long and distinguished history and widely accepted in fields such as fluid dynamics solving nonlinear, multidimensional, and dynamic systems. The extensive literature on its applications encompasses diverse range of domains, including musical acoustics [10]. A recent contribution an FDTD simulator tailored for modeling planar
45 and electromagnetics [46]. FDTD is particularly effective in
by Lee et al. 47 introduces StringFDTD-Towe damped stiff strings akin Equation 3] Leveraging PyToorch C++ extension, BtrigFDDDDTTore facilitates FDTD computations on both CPUs and GPUs. However, its current iteration lacks support for gradient backpropagation through the FDTD module, leaving room for enhancement, particularly in optimizing the gradient computation process, which is hindered by the substantial number of temporal recursions involved (evident by the large N, in O(,,.Ne) of Table 1..
Modal Synthesis. As a more efficient approach to solve the nonlinear wave equations, the modal synthesis (48 decomposes the complex dynamics into contributions from a set of modes, whose spatial bases are eigenfunctions of the pertinent problem. Each mode exhibits distinct oscillations at complex frequencies, contingent upon the boundary conditions. For problems with real-valued parameters, these complex frequencies occur in conjugate pairs, and the "mode" is thus defined as the pair of such eigenfunctions and frequencies [10]. Modal synthesis involves two primary steps. Initially. in an offline phase (also labeled as Pre-computation' in Table I), modal shapes and frequencies are discerned from the PDE system, considering both boundary and initial conditions. This information is encapsulated in what is known as a shape matrix. Subsequently, the solution is derived by combining the modal functions, each progressing at its natural frequency. Pre-computation requires O (NNm) of recursions because Nm shape matrices need to be computed, but a typical Nm. typically hundreds to thousands of times less for N,. Modal synthesis after this off-line process is very efficient as it allows us to obtain a solution for given r and & without any recursion, but it is clear that the range of solutions that can be covered is bounded in that it relies on the ansatz uxx t)= (t) for the separation of variables.
2.2 Differentiable Digital Signal Processing
In the field of neural networks, numerous audio researchers have been engaged in the development of techniques that leverage the ease of automatic gradient backpropagation in neural networks for the purpose of audio parameter estimation. To address the challenge of estimating the latent parameters of a sound, some approaches implement the synthesis part as-is using automatic differentiation package 20 (7) so that the gradient can back-propagate through it update the parameters directly, while the majority of approaches train neural networks to estimate the parameters in an auto-encoder framework 51] As one of the most seminal studies of the latter approach, DDSP is widely used to efficiently synthesize nonlinear and dynamic sounds. Based on the spectral modeling synthesis framework, the time-domain signal is modeled via short-time Fourier transforms (STFTs) divided into deterministic (harmonic) and stochastic (noisy) parts to synthesize the sound. As the causality of STFT frames modeled through gated recurrent units (GRUs), DDSP requires as many recursions as Nr, the number of frames. The harmonics of a DDSP are given by an integer multiple of the fundamental frequency (fu), and the noise is synthesized from filtered noise. These DDSPs, while still remarkable advancement, have strong structural constraints on the deterministic part to capture enough perceptually rich tones such as inharmonicity due to stiffness or phantom partials due to nonlinearity. A study by Renault et al. 2 also points this out, and uses the parametric model (26] for inharmonicity and detune, but there is room for improvement as is an approximation model that relies on instrument- -psccfi modifiers rather than reflecting stiffness physics.
16 22

---

Initial Condition
Tension. Time
Input
Feature
(Plucked String)
Stifnness
Parameter Encoder
Noise Decoder
Damping (T60)
MLP
Pitch
Mode Estimator
Feature
Space
Inharmonic
AMl Tarh FMt OLU
Pitch
Mode Amplitude
Skeleton
FC
FM
Mode Frequency
Output
(a) Network architecture DMSP
(b) AM/FM Block
Figure 3: Network architecture. DMSP synthesizes pitch skeleton with inharmonic structure, drawing upon overtones derived from the modes of the string. The modes can either be derived directly using the modal decomposition (DMSP-Hybrid, the hybrid of DMSP and Modal), or using the neural network trained to estimate the modes (DMSP, the sully-neural-networy method), Yet, relying solely on modal frequencies and corresponding shape functions delineates linear solution, which falls short of capturing the nuances of nonlinear string motion. To address this, DMSP introduces FM and AM blocks to modulate the modes of the linear solution. This modulation process enables DMSP to estimate the pitch skeleton of the nonlinear solution. Consequently, the output waveform is synthesized through the spectral modeling pipeline, incorporating both (in)harmonic components and the filtered noise.
3 Differentiable Modal Synthesis for Physical Modeling (DMSP)
This section introduces a novel differentiable nonlinear string sound synthesizer, Table provides a summary of the methods discussed. While modal synthesis stands out for its efficiency, it is solely applicable to linear models and necessitates pre-computation to determine the number modes denoted as Nm. On the other hand, FDTD computes highly nonlinear and dynamic solutions but demands a substantial computational load due to iterative updates across both temporal (N₂) and spatial (N₂,) samples. In contrast, differentiable audio processing methods offer efficient nonlinear sound synthesis, typically leveraging smaller number of frames (N₂)) compared to the total time samples (N,). However, they often lack physical controllability. In this regard, we propose DMSP, which approximates the solution of Equution Jefficiently by leveraging [Equation 2 in the form of neural networks. Figure provides a visual depiction of the DMSP.
3.1 Problem Statement
The objective of this study establish mapping from the parameter space to the solution space,
utilizing a finite set of observations comprising parameter-solution pairs from this mapping. We
delineate this objective as follows: Consider the partial differential equation depicted in Equation applicable for (z, t) within 52 x [0, oo), with clamped boundary conditions as specified in Equation 4 where represents a bounded domain in N= We assume that the solution u. Ω x [0, oc) + R
resides within the Banach space U. For given PDE parameter P and initial condition uo E Ut
let S: P - l4 denote nonlinear map, specifically, the FDTD numerical solver tailored to the context of this study. Assume that we are provided with observations {p(t), (t)) 1. independent and identically distributed (i.i.d.) samples drawn from probability measure supported on P. and (i) = S( potentially contains noise. Our goal is to construct an approximation of S denoted S P U, and select parameters 0" E Ne such that Sn- S. Leveraging Sg, one can compute the solution á = So(p) corresponding to a new parameter p E P. By specifying values for and one can then either synthesize the sound of the string picked-up (also referred as read-out) at specific location Zg as ú(z., t), or simulate the motion of the string by concatenating ū(n, t) across all x E Ω. In practice Ω and [0, oo) are bounded and discretized to form an evenly distributed spatio-temporal grid as N. x N: C Ω distributed space according to the length of the equally spaced intervals T/NN and the temporal grid samples uniformly distributed according the frequency of fixed audio sampling rate.
Ji=1 where p(²) comprises
x [0, oo). where the spatial grid samples are uniformly
5

---

3.2 Network Architecture
Parameter Encoder. To effectively capture material features inherent the PDE parameter values, the parameter encoder leverages a random Fourier feature (RFF) layer 552 (33. Given T60 frequencies fo and their corresponding times (44 for i = 1, 2, the frequency-
damping coefficients do (and ø1, if applicable) are derived using [Equation 24 The frequeney-independent damping factor
is computed explicitly multiplied by the mode amplitudes. All PDE parameters p= = {K, a, 0o. 01) I c P are encoded into a feature vector h E Ixxd with a Fourier embedding size of d = 256.
AM and FM Blocks. As illustrated in Figure 3 DMSP employs amplitude modulation (AM) and frequency modulation (FM) modules to modulate the mode frequencies and amplitudes of the linear solution, as depicted in [Equation 2, synthesize the solution of the nonlinear wave described in [quation 3 We utilize two multilayer perceptrons (MLPs) for the modulation layers. Although simpler and perhaps more conventional choice of architecture would be GRU, to the decoder architecture of DDSP [16], we choose MLPs for their slightly better empirical results. It's noteworthy that DDSP decodes the sinusoidal frequency envelope with fixed frequency values. In contrast, DMSP decodes both the envelope and the frequency values independently, employing two distinct MLP blocks, namely AM and FM.
Mode Estimator. As detailed in (uussecttion 2.1, determining allowed values for mode frequencies and amplitudes, corresponding specifie initial and boundary conditions of the string, typically involves a root-finding process conducted offline, While these numerical solvers offer high accuracy up specified iterative threshold, they necessitate pre-computation, as illustrated in TTable I The mode estimator module within DMSP estimates the modes from the initial condition using an MLP. The initial condition is parameterized by a pluck position Ps: and its peak amplitude Pa. Subsequently, the physical properties K, 7/, To. and are encoded using random Fourier feature (RFF) layer. The mode frequencies and amplitudes are then estimated by the MLP, followed by the application of suitable scaling activations. It's pertinent to note that the mode estimator operates independently and trained separately from the other modules. During training, the ground truth modes (computed using the modal decomposition method) are fed into the AM and FM blocks, ensuring accurate synthesis while training the synthesis part of the model.
3.3 Loss Function
We employ a combination of four loss terms: (I) waveform & loss (C1) that measures the L, dis- crepancy between the synthesized waveform and the ground truth waveform, (2) Multi-scale spectral (MSS) loss that captures spectral differences across multiple scales, ensuring fidelity in spectral representation, (3) Pitch loss (c),) that penalizes deviations from the ground truth fundamental frequency (fn). and (4) Mode loss (C,,), which measures the L, distance of the mode frequency and mode amplitude from the output of the mode estimator (if applicable) and the mode frequency and mode amplitude obtained via modal decomposition. MSS loss has been adopted as a metric for reconstruction in most neural net-based synthesis techniques [6 21 Concerning that measuring MSS with magnitudes is not phase-sensitive by definition, we employ the Li loss in the waveform train the causality in spatio-temporal wave propagations. Challenges in optimizing the frequency parameters of sinusoidal oscillators via gradient descent over the spectral loss functions, due to the non-convex nature of the optimization problem, have been highlighted in various studies [54] 55). Damped sinusoids offer a workaround for the issue of non-convexity concerning frequeecyy param- eters [55]. or alternative metrics are proposed to mitigate the risk of falling into bad local minima [6 57]. We adopt a parameter regression joint training strategy, akin to the pre-training phase of the work by Engel et al. (88] Among the output mode frequencies, we train the model to match one mode frequency component (denoted by fo) to match the fundamental frequency of the target FDTD SomD-simmlated audio (fo) annotated using CREPE (5 as C,, = Ilfo - foll1.
'It is worth noting that there are some subtle differences from the AM/FM techniques commonly used in sound synthesis and the usage of the terminology in this paper. Relying on terminology commonly used in AM/FM technique, AM/FM this paper can described analogically as a carrier sinusoid whose center frequency and amplitude are determined by modal decomposition and varied by a non- non-sinusoidal modulator. While typical AM/FM synthesis also uses non -inusooooi modulators, the intent of this paper is different in that it borrows these for the purpose of limiting the undesired periodicity in those modulators.

---

Table 2: Comparison between the baselines and the proposed models.
Network architecture
Training configuration
Method
Center frequency
AM block
FM block
L1
MSS
fo
Lm
Modal
Mode frequency
x
X
N/A
N/A
N/A
N/A
DDSPish-XIM
Integer multiples
/
X
/
X
DDSPish
Integer multiples
/
/
/
/
X
X
DMSP-Hybrid DMSP
Mode frequency
/
V
V
X
Learnt estimates
V
V
4
Experiments
4.1
Experimental Setup
Dataset. We use the StringFDTD-Tomee Torch [47], the open-source nonlinear string simulator, to compute the solution of the Eqqaaion i a temporal sampling rate of 48 kHz and a spatial sampling rate. The solution is upsampled to a spatial resolution of 256 using a bivariate spline approximation over a rectangular spatio-temporal mesh upto the 5th order degree. We simulate 10263 different strings by randomly augmenting the material properties, e.g., K, a, To. and Ø1, with various plucking profiles ug. The simulation results a total amount of 729.8 hours of wave files that corresponds to the 1-sec string sounds picked up at 256 different positions for each string. For the test data, 715 strings are newly synthesized with the parameters sampled in i.i.d The test data consists of 336 linear (a = 1) and 379 nonlinear (a > 1) strings. each of which has 256 spatial grid size, resulting in approximately 50 hours of wave files. [Table 5| Speeeiiiss the range of the sampled PDE parameters.
Table 2 compares the major differences between the baselines and the proposed models. As the first attempt to tackle the task of neural audio synthesis for dynamic physical properties, compare our proposed model to three other models. We consider three baselines, namely: Modal and DDSPish, and DDSPish-XIM Modal synthesis is the linear wave solution as in Eqquaion I] where the modal frequencies and the shape functions are pre-computed. DDSPish is a neural network based on the harmonic plus noise model, similar to DDSP. Yet, this -ish suffix emphasizes that this model is different from the DDSP model [6]. DDSPish does not have a reverb module but instead adds frequency modulation, and most notably has a parameter encoder that allows generating sounds from physical parameters. Please see Appendix Doro more details of the baselines.
Baselines.
Evaluation Metrics. We report results on three metrics, signal-distortion-ratoo (SDR), scale-invariant signal-distortionn-atio (SI-SDR) (60), multi-scale spectral (MSS) distance, and the pitch difference in Hz. As with the problem statement, we consider the FDTDD-immulated results the ground truth (GT). Both SDR and SI-SDR estimate the distortions on a spatiotemporal grid and are very strict about scoring out-of-phase cases, by directly comparing the the estimated displacements to the FDTD results without any specific transformations or interpolations. Given that the magnitudes of the estimated displacements typically distribute within the small range (approximately between =0.02; depends on the pluck amplitude Pa. see seele 5). we compare both with scale normalization (SI-SDR) and without (SDR). The MSS metric for the evaluation computed using the short-time Fourier transformation (STFT) with three scales of Fast Fourier Transform (FFT) points: 1024, 512, and 256, with window length equal to the FFT point for each, and hop length equal to a quarter that. For each scale, the STFT magnitude is compared on both linear and log scales, weighted by 2.0 and 0.5, respectively. This choice of weighting to help scale the loss computation with MSS and should not have significant impact performance, but is important to note for absolute comparisons of MSS scores. The Pitch metric is computed the el norm of the difference between fo and fo-
4.2 Results
Differentiable Sound Synthesis. The efficacy of DMSP is studied as shown in Table 3 First, the modal synthesis method calculates linear solution for a damped stiff string, and the mode for the linear test set is calculated offline. The discrepancy between the outcomes of modal synthesis and FDTD in the linear case can be attributed to the absence of frequency-dependdnt damping in Eqqution In other words, the frequency information serves as an oracle for the Modal, as evidenced by its lowest pitch score, but the decay of amplitude over time not sufficiently modeled, which is where the remaining models demonstrate superior performance. Considering the DDSPish models,
7

---

Table 3: Synthesis Results
Linear (a 1)
Nonlinear (a > 1)
Model
SI-SDR
SDR
MSS
Pitch
SI-SDR (dB, f)
SDR
MSS
Pitch
(dB, f)
(dB, f)
(dB,
(Hz,
(dB, f)
(dB, 4
(Hz, 4)
Modal
-3.191
0.681
18.449
0.420
16.611
-1900
17.254
2.316
DDSPish
39.4788
-2.998
11.047
5.518
25,9511
-2.102
9.745
3.306
DDSPish-XPM
46.609
-2.257
10.911
11.304
46.858
-2.27
10.299
14.013
DMSP-Hybrid
-2.844
1.496
12.525
0.792
15.670
16.455
4.772
1.027
DMSP
-22.298
-2,000
12.504
1.717
100315
0.221
5.656
1.437
(a) Ps 0.1
(b) p. = 0.2
(c) P. 0.3
(d) p. 0.4
Figure 4: Visualization of the string displacement over time (horizontal) and space (vertical), For different initial conditions, the results synthesized by DMSP are shown as solid black lines and those simulated by FDTD as dashed gray lines.
the MSS is approximately 1. dB ahead of the DMSPs, but the difference can reach up to 44 dB in the case of the SI-SDR. It is worth noting that is uniformly sampled within the range (1, 25) for the training data, falling into the category of the nonlinear strings. Information about the ranges for sampling PDE parameters is in Table In the case of the nonlinear modal solution and the FDTD solution becomes larger. On the other hand, DDSPish models, which are based on spectral modeling synthesis but learned without using any modal information, show the lowest performance. The superiority of DMSP is even more pronounced in the nonlinear case. While Modal, which can only cover the solution the linear case, shows attenuated performance, the DMSPs demonstrate the best performance in all metrics. In particular, DMSP- precomputes the mode frequency like Modal, performs FM and AM for the nonlinear solution, showing an SI-SDR improvement of nearly 31 dB and an MSS improvement of nearly 13 dB over Modal. DMSP, which estimates the mode as a neural network without the pre-computation step, also outperforms Modal all metrics for the nonlinear case. The primary reason for the performance discrepancy between the nonlinear case and the linear case for DMSP is that the training data is rarely precisely equal to 1 in the distribution when it is sampled.
set, the difference between the
which
Controllable Physical Simulation. The quantitative scores for various physical condition parameters are visualized in Figure 5 Trends show how the results of Modal synthesis and DMSP vary for different pickup positions (æ:), stiffness (s), tension (o), pluck amplitude (pa), and pluck position (p.). Of these, and Pa in particular are known to increase the nonlinearity of the string as they increase magnitude, which can seen an
SASS 5005 SSV 19t1
107
by the lower Modal synthesis scores. For DMSP, overall improvement in the score, with lower propensity for nonlinearity. Figure 4/depicts the simulated state of the string as the pluck position in the initial condition is varied. The results synthesized by DMSP initial condition, similar to the results simulated by FDTD. The vibration propagating through time along the string exhibits distinct behavior contingent upon the initial condition. FDTD employs a recursive caleulation of displacement, necessitating some iterations equal to the number of samples the audio sampling rate. In contrast, DMSP capable of obtaining the desired displacement in both time and space simultaneously.
Medal
DMSD
Figure 5: Objective scores over the change of physical parameters.
reconstruct very accurate

---

(a) Modal
(b) DDSPish-XFM
(c) DDSPish
(d) DMSP
(c) DMSP-Hybrid (f) FDTD (GT)
Figure 6: Spectrogram of the synthesized samples on the test set.
(a) Modal
(b) DDSPish-XPM
(c) DDSPish
(d) DMSP
(e) DMSP-Hybrid (f) FDTD (GT)
Figure 7: Spectrograms and state samples of the synthesized samples on the test set. For the spectrograms shown in the first column, the intensity of the frequency (vertical axis) component for time (horizontal axis) is expressed as brightness, and for the states shown in the second column, the displacement (vertical axis) for space (horizontal axis).
String Sound Synthesis. Spectrograms of the samples are visualized in Figure 6 Figure For the spectrograms, the instantaneous frequencies are identified in a rainbow color map, where the color intensities represent the logarithmic magnitude of the power spectra. Observing from Figure 6 the FDTD-simee EDTD-simulated spectrogram clearly shows pitch glide and phantom partials at the beginning of the pluck. In contrast, modal synthesis methods that model linear solutions do not show these nonlinear characteristics. The DDSPish-XFM model employs harmonic pitch skeleton comprising integer multiples of fo. thereby precluding the inharmoicities resulting from stiffness. The DDSPish model demonstrates enhanced mode estimation capabilities through learned FM, which modulates the harmonic pitch skeleton to be inharmonic. However, there is scope for further improvement in frequency estimation, particularly in instances where the FM learning process is unstable for high frequencies. the other hand, the DMSP model, which estimates the mode frequency and amplitude from ug. shows improved pitch skeleton and stable frequency estimation. The DMSP-Hybue model, which learns AM and FM the sinusoidal oscillators of modal solution that requires mode precomputation, shows the most similar results to FDTD.
shows similar trend for the spectrograms. difference that is not apparent in the spectrograms, but is very important for physical modeling: It the measure displacement to space, namely the state plot. The timestep for the state plot can be arbitrary, but we show the initial state for ease of comparison. For methods that do not have separate method for accurately predicting the mode information, unlike DMSP, difficulties can be found in accurately predicting the state. This partially explains the SI-SDR scores for DDSPish and DDSPish-XPM in Table 3 Considering that the major difference between DMSP and DDSPish is whether the input frequency of the FM block is the mode frequency or an integer multiple, it can be inferred that accurately estimating the mode frequency is critical from a physical modeling perspective to fit the displacement of the strings over time.
Figure 7
this example, however, we reveal a key
Ablation Study, The ablation study on the choice of the training loss functions is presented in Table Overall, for linear strings, the performance does not vary much depending on which loss is used, especially for pitch. This is due a gating applied the FM block, which designed to prevent FM from occurring when is 1. More specifically, we apply gating, e.g., tanh(a 1). in such way that frequency modulation only occurs when value of a deviates from 1, which can directly affect pitch, and otherwise forces the mode estimation result to be used as is. This is the main factor that determines the nonlinearity of the string. For AM, there no such masking depending on a, so the remaining metrics except pitch do vary. For nonlinear data, the training results vary depending the

---

Table 4: Ablation Study
Linear
Nonlinear
Model
SI-SDR
SDR
MSS
Pitch
SI-SDR
SDR
MSS
Pitch
(dB, f)
(dB, t)
(dB, 4)
(Hz, 4)
(dB, 1)
(dB, f)
(dB, 4
(Hz, 4
DMSP-Hybrri
-2844
1.496
12.525
0.792
15.670
16.455
4.772
1.027
-2.119
0.774
13.487
0.792
-5.418
1.509
8.983
2.653
DMSP
-22.298
-2.000
12.504
1.717
-10.315
0.221
5.656
1.437
w.o. fo
-21.351
-2.699
13.482
1.717
-16.435
-1.074
9.060
2.922
Displacement
Spectrogram
Displacerment
Spectrogram
FDTD
DMSP FDTD
DMSP Modal
Medall
Bower DMSP
Moshal
Ernor Mocial
Figure 8: Simulated string state visualization.
loss design. In particular, for all metrics, using C,, loss significantly improved performance over not using This difference especially noticeable in the MSS scores between DMSP-Hybrid w.o. f. and DMSP, where DMSP even gives better results over the model that uses precomputed mode frequencies for nonlinear MSS scores if L, applied. This result reaffirms the validity of the FM block for nonlinear strings in terms of predicting dynamically varying frequencies such as pitch glide and shows that Ly, loss is the loss function to train it effectively.
Motion Synthesis. The main advantage of DMSP, compared to the existing sound synthesis models, that it can synthesize not only sound but also motion, which is one of the main characteristics of physical modeling techniques. In particular, the DMSP can visualize the corresponding string motion as video, although the receptive field and computational complexity required obtain solution u(x, () for single spatio-temporal point is of order 1, as shown in Table 1, when these solutions are pooled for a given x E Ω and & [0, 1), the corresponding string motion can visualized as video. Figure 8 visualizes the resulting transverse displacement of the string over time (horizontal axis) and space (vertical axis). The transverse displacement of the FDTD coupled to the longitudinal motion, which is why it differs from the Modal synthesis output, which synthesizes the motion of linear damped stiff string. The results output by DMSP show improved accuracy.
Conclusion
We present novel neural network-based method that efficiently simulates plucked string motions. Our differentiable modal synthesis for physical modeling (DMSP) can simulate a dynamic nonlinear string motion by synthesizing the sound using the physical properties of the string. It is efficient approximation of existing physical modeling methods. We demonstrate the efficacy of training the neural network using mode frequency information by extending the DDSP with a modal synthesis pipeline. This opens the door to a new field of differentiable audio signal processing, extending it the field of physical modeling for musical sound synthesis. While the proposed method offers control over several physical parameters of a musical instrument, it still faces limitations in terms of generalizing to physical parameters and sounds real-world measurements. This study paves the way for future research in this area. To the best of our knowledge, this is the first study to simultaneously synthesize sound and motion from the properties stringed instrument.
10

---

Acknowledgments
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) INO. RS-2023-002194223 and partly by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) INO.RS-225 2SS-2022-1122111 113111 Artificial Intelligence Graduate School Program (Seoul National University)].
References
[1] John William Strutt Baron Rayleigh. The theory of sound, volume 2. Macmillan, 1896.
[2] William Fishbum Donkin. Acoustics: Theoretical. Part 1. Clarendon Press, 1884.
[3] Harvey Fletcher. Normal vibration frequencies of stiff piano string. The Journal of the Acoustical Society of America, 36(1):203-209 1964.
[4] Pierre Michel Ruiz. technique for simulating the vibration of strings with a digital computer. PhD thesis, University of Illinois at Urbana-Chane Jobban-Chaampain, 1970.
[5] Lejaren Hiller and Pieme Ruiz. Synthesizing musical sounds by solving the wave equation for vibrating objects: Part 1. Journal of the Audio Engineering Society, 19(6), 1971.
[6] Diemo Schwarz. Corpus-based concatenative synthesis. IEEE signal processing magazine, 24 (2):92-104, 2007.
[7] Xavier Serra and Julius Smith. Spectral modeling synthesis: A sound analysis/synthesi system based on a deterministic plus stochastic decomposition. Computer Music Journal, 14(4):12-24, 1990.
[8] Julius O Smith III. Physical audio signal processing: For virtual musical instruments and audio effects. 2010.
[9] Alfred Fettweis. Wave digital filters: Theory and practice. Proceedings of the IEEE, 74(2): 270-327, 1986.
[10] Stefan Bilbao. Numerical sound synthesis: finite difference schemes and simulation in musical acoustics. John Wiley & Sons, 2009.
[11] Ben Hayes, Jordie Shier, György Fazekas, Andrew McPherson, and Charalampos Saitis, A review of differentiable digital signal processing for music and speech synthesis. Frontiers in Signal Processing, 3: 2221100, 2024.
[12] Aäron van den Oord, Sander Dieleman, Hciga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Arxiv, 2016. URL https: //attps://rrivv
abs//60s. 03499
[13] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Interna- tional Conference on Learning Representations, 2018.
[14] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. In International Conference on Learning Representations, 2018.
[15] Jean-Mare Valin and Jan Skoglund. Lpenet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5891-5895. IEEE, 2019.
[16] Jesse Engel, Chenjie Gu, Adam Roberts, al. Ddsp: Differentiable digital signal processing. In International Conference on Learning Representations, 2019.
[17] Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for statistical parametric speech synthesis. IEE.F/ACM Transactions on Audio, Speech, and Language Processing, 28:402-415, 2019.

---

[18] Ben Hayes, Charalampos Saitis, and György Fazekas. Neural waveshaping synthesis. arXiv preprint arXiv:2107.05005, 2021.
[19] Franco Caspe, Andrew McPherson, and Mark Sandler. DDX7: Differentiable FM Synthesis of Musical Instrument Sounds. Proceedings the 23rd International Society for Music Information Retrieval Conference, 2022.
[20] David Braun. DX7-JAX, November 2023.
[21] Yusong Wu, Ethan Manilow, Yi Deng, Rigel Swavely, Kyle Kastner, Tim Cooijmans, Aaron Courville, Cheng-Zhi Anna Huang, and Jesse Engel. Midi-ddsp: Detailed control musical per- formance via hierarchical modeling. In International Conference on Learning Representations, 2021.
[22] Tleong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervieed representations. Advances in Neural Information Processing Systems, 34:16251-16266 2021.
[23] Hyeong-Scok Choi, Jinhyeok Yang, Juhcon Lee, and Hyeongju Kim. Nansy++: Unified voice synthesis with neural analysis and synthesis. In The Eleventh International Conference Learning Representations, 2022.
[24] Adrián Barahona-Riss and Tom Collins. Noisebandnet: controllable time-narying neural synthesis of sound effects using filterbanks. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:1573-155 1585, 2024.
[25] Lenny Renault, Rémi Mignot, and Axel Roebel. Differentiable piano model for midi-to-audio performance synthesis. In 25th International Conference Digital Audio Effects (DAFx20in22), 2022.
[26] François Rigaud, Bertrand David, and Laurent Daudet. A parametric model of piano tuning. In Proc. of the 14th Int. Conf. on Digital Audio Effects (DAFx-11), pages 393-399, 2011.
[27] David Henningsson and FD Team. Fluidsynth real-time and thread safety challenges. In Proceedings of the 9th International Linux Audio Conference, Maynooth University, Ireland, pages 123-128, 2011.
[28] Balazs Bank and Juliette Chabassier. Model-based digital pianos: from physics to sound synthesis. IEEE Signal Processing Magazine, 36(1): 2018
[29] Sebastian Schlecht, Julian Parker, Maximilian Schäfer, and Rudolf Rabenstein. Physical modeling using recurrent neural networks with fast convolutional layers. In International Conference on Digital Audio Effects, pages 138-145. DAFx, 2022.
[30] Doug James, Jernej Barbič, and Dinesh K Pai. Precomputed acoustic transfer: output-sensitive. accurate sound generation for geometrically complex vibration sources. ACM Transactions on Graphics (TOG), 25(3):987-995, 2006.
[31] Jui-Hsien Wang and Doug L James. Kleinpat: optimal mode conflation for time-domain precomputation of acoustic transfer. ACM Trans. Graph.. 38(4):122-) 2019.
[32] James F O'Brien, Chen Shen, and Christine M Gatchalian. Synthesizing sounds from rigid- body simulations. In Proceedings of the 2002 ACM SIGGRAPPH/Enngraphic symposium on Computer animation, pages 175-181, 2002.
[33] Xutong Jin, Sheng Li, Tianshu Qu, Dinesh Manocha, and Guoping Wang. Deep- real- time impact sound synthesis for arbitrary shapes. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1171-1179, 2020.
[34] Xutong Jin, Sheng Li, Guoping Wang, and Dinesh Manocha. Neuralsound: learning-based modal sound synthesis with acoustic transfer. ACM Transactions on Graphics (TOG), 41(4): 1-15, 2022.
12

---

[35] Rodrigo Diaz, Ben Hayes, Charalampos Saitis, György Fazekas, and Mark Sandler. Rigid-body sound synthesis with differentiable modal resonators. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEFE, 2023.
[36] Gustav Kirchhoff. Vorlesungen über mechanik, volume 1. BG Teubner, 1897.
[37] GF Carrier. On the non-linear vibration problem of the elastic string, Quarterly of applied mathematics, 3(2), 1945.
[38] Stefan Bilbao and Michele Ducceschi. Models of musical string vibration. Acoustical Science and Technology, 2023.
[39] Philip McCord Morse and K Uno Ingard. Theoretical acoustics. Princeton university press, 1986.
[40] GV Anand. Large-amplitude damped free vibration of a stretched string. The Journal of the Acoustical Society of America, 45(5):1089-1099 1969.
[41] Stefan Bilbao. Conservative numerical methods for nonlinear strings. The Journal of the Acoustical Society of America, 118(5):331-33222 2005.
[42] Stefan Bilbao. Energy-conserving finite difference schemes for tension-modulated strings. In 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume pages iv-iv. IEEE, 2004.
[43] Esteban Maestre, Gary P Scavone, and Julius O Smith. Joint modeling of bridge admittance and body radiativity for efficient synthesis of string instrument sound by digital waveguides. IEEE/ACM Transactions on Audio, Speech, and Language SringgeePPrcceee Processing. 25(5): 128- 139, 2017.
[44] Lutz Trautmann and Rudolf Rabenstein. Multirate simulations of string vibrations including nonlinear fret-string interactions using the functional transformation method. EURASIP Journal on Advances in Signal Processing, 2004:1-15, 115, 2004.
[45] Parviz. Moin and Krishnan Mahesh. Direct numerical simulation: tool in turbulence research. Annual review fluid mechanics, 30(1):539-577, 1998.
[46] Allen Taflove, Ardavan Oskooi, and Steven Johnson. Advances FDTD computational electrodynamics: photonics and namotechnology. Artech house, 2013.
[47] Jin Woo Lee, Min Jun Choi, and Kyogu Lee. String sound synthesizer on gpu-accelerated finite difference scheme. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1491-1495. IEEE, 2024.
[48] Jean-Marie Adrien. The missing link: Modal synthesis. In Representations of musical signals, pages 269-298. 1991.
[49] Meane Adrien and Xavier Rodet. Physical models of instruments: A modular approach, application to strings. In ICMC, 1985.
[50] Joseph Derek Morrison and Jean-Marie Adrien. Mosaic: A framework for modal synthesis. Computer Music Journal, 17(1):45-5,, 1993.
[51] Sungho Lee, Hyeong-Seok Choi, and Kyogu Lee. Differentiable artificial reverberation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 30:2541-2556, 2022.
[52] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[53] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kein Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
[54] Joseph Turian and Max Henry. I'm sorry for your loss: Spectrally-based audio distances are bad at pitch. In "I Can't 't Believe It's Not Netuer!"NeurIPS 2020 workshop, 2020.
13

---

[55] Ben Hayes, Charalampos Saitis, and György Fazekas. Sinusoidal frequency estimation by gradient descent. In ICASSP 2023-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[56] Bernardo Torres, Geoffroy Peeters, and Gaël Richard. Unsupervised harmonic parameter estimation using differentiable dsp and spectral optimal transport. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1176-1180. IEEE, 2024.
[57] Simon Schwär and Meinard Müller. Multi-scale spectral loss revisited. IEEE Signal Processing Letters, 30:1712-1716, 2023.
[58] Jesse Engel, Rigel Swavely, Lamtharn Hanoi Hantrakul, Adam Roberts, and Curtis Hawthorne. Self-supervised pitch detection by inverse audio synthesis. In ICML 2020 Workshop on Self- supervision in Audio and Speech, 2020.
[59] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe: convolutional representation for pitch estimation. In ICASSP, pages 161-165. IEEE, 2018.
[60] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 626-630. IEEE, 2019.
[61] Balázs Bank and László Sujbert. Physics-based sound synthesis of string instruments including geometric nonlinearities, 2006.
[62] Balaz Bank and Laszlo Sujbert. A piano model including longitudinal string vibrations. In Proceedings of the Digital Audio Effects Conference, pages 89 -4, 2004.
[63] Lutz Trautmann and Rudolf Rabenstein. Digital sound synthesis by physical modeling using the functional transformation method. Springer Science & Business Media, 2012.
[64] Maximilian Schäfer, Petr Frenstátský, and Rudolf Rabenstein. A physical string model with adjustable boundary conditions. In 19th International Conference on Digital Audio Effects (DAFx-16), Brno, Czech Republic, pages 159-166, 2016.
[65] Rudolf Rabenstein, Tilman Koch, and Christian Popp. Tubular bells: A physical and algorithmic model. IEEE transactions audio, speech, and language processing, 18(4):881-890, 2009.
[66] Maximilian Schäfer, Sebastian Schlecht, and Rudolf Rabenstein. A string in a room: Mixed- dimensional transfer function models for sound synthesis. In International Conference on Digital Audio Effects, pages 187-194. DAFx, 2020.
14

---

A Proof of the Linear Damped Stiff String Solution
Proposition 1. A solution to the damped linear stiff string model Equation 1 Twith clamped boundary
condition u(+I/2211) 1) = Uz (= = O and the initial condition given as u(x, 0) = uo (x) and Ozu(2, 0) 0 can be expressed
u () Σ
(2(x)Ta(()
n=1
sin L/2 sinh pnt
COs |4, L1, cosh 1/x
(x) C1
sin pnI
+ cos fn22 cosh L/2
sinh V/ L/2
(t) = e dant cos (VuAn² + 12,2 0 t
Proof. As it is hinted in Equation 2a the procedure to derive Equation 26 and Eqqatiin 2c uses the method of separation of variables. The derivation consists of three main steps that start by trying the ansatz ((z,t) (t)) Substituting this ansatz into the Equation gives
X"
(4)
T"
200 T
(S)
X
X
T
with E R as the separation constant.
1. Solving for T.
T" + T0 =0
(6)
Roots of the characteristic polynomial this equation are B1 -00 Vog solutions are available:
Three
Overdamping > T== A
2-t
Critical damping (o) = =) (A1 + A2t)e ogt
Underdamping (aî < 5): Rewrite the roots as 34 = = + iw and B_ -00
with w = 5²2 o a = v ob. then
T = c out out (Aieatt + A2t)
The initial condition (z,( 0) = implies Ag = 0 yielding the real solution of the form T' Aie oot cos wt. This implies that the only valid root is B4 -00 iw, hence enough to consider the - case only, in the Equation 3)
This work only considers the underdamped case where the a is sufficiently small, as should be in reasonable string model for any musical purposes.
2. Solving for X.
(
X"
X
(8)
x²
K²
Substitute = 72/2x² and 5² 2/22. The roots of the characteristic polynomial for this equation are + + + Vl2 where 1 v12 m < 0. Therefore the solution is rewritten
as
X(x)= = Bj sin + sinh or
B3 cos
B₁ cosh vr
odd function
function
where = v + m and v V V12 + m + I. By applying the boundary condition ([L/2,t) to the even and odd functions each, then obtain
sin 4
cos /
B2 =
sinhv B1 and B1 = cosh // 1
Bg
(10)
reducing the solution as
sin 4 sinh v4
cos 1
X(x) = B1 sin pz
sinh vx
+ B3
cos pa
cosh vr
(11)
cosh
15

---

3. Finding allowed values that satisfy the conditions. Substituting the ansatz by Equation 7 Equation 11 summarizes the solution as
( 12 oz.
u(x, = = cos
(12)
sin 2 sinh V
co8 1
x
Ci sin puzz
sinh wz
+
Cos par
cosh
cosh /
As mentioned earlier, the coefficients 6 = A, and C2 = B3 are determined by the allowed values that satisfy the initial condition Σ Ex=1 (x) uu(x). Yet, this is usually preceded by obtaining the μ and v values. To determine the allowed values of μ. apply the boundary condition 2 t)) uy (L/2, 0 the even function that gives
I. = -B cosh v and BBg sin 2 = ----BBB BB By cos
I.
(13)
2
alternatively,
L tan 2 = - - tanh L
(14)
Similarly, applying the same boundary condition to the odd function gives
By sin / I. B4 sinh
I.
I. = BB cosh v and HBy cos cosH
(15)
or equivalently,
L lan 11 p tanh 2
L
(16)
By finding the values of and v = Vμ² + 2/ that satisfy the Equation and Equation 16 Un and P/, one can determine the allowed values of m, (M.. + +1)² 2)²- - and Sn v V
Following these three steps gives the ri-hh mode oscillation of u(x 1) X, (t) where
sin pn L/2
cos p4 L/2 cosh 1/
(x) = sin pn.
sinh Vn + C2 COs
Un (0084
(17)
sinh L/2
cosh L/2
and
(t( = - cos (Vuti2 + 22: og
(18)
Hence, the modal solution is expressed as a superposition of modes, where the infinite summation as Equation 2a finalizes the modal expression of the modal solution.
B Nonlinear Damped Stiff String Vibration
The Kirchhoff-Carrier model stands out one of the simplest representations of nonlinear distributed strings, as it effectively captures the pitch glide effect and can be easily simulated using relatively straightforward finite difference schemes. However, it is also known that only the transverse motion explicitly accounted for, so the longitudinal motion is averaged across the length of the string. Obviously, there reports that this approximation may suffice under specific conditions [61]. On the other hand, it is also reported that the interaction between longitudinal and transverse motion can result in generating rich timbres with perceptually crucial effects such as phantom partials 38 A comprehensive model of string vibration, which incorporates both longitudinal and transverse motion within a single plane (in dimensional form), is outlined follows:
O
pAduu E.A.. (FAA a
(19a)
a(O.u)
04
pAduS = EAA.. - (EA To)a,
(19b)
a(a.c)
The constants and To are the material density and tension, respectively. In order to model the stiffness, Young's modulus E and the cross-scctional area A are introduced.
16

---

The function D. which nonlinearly connects the two equations, is defined as follows:
+ / 02 +,,u²
(20)
Note that the final term -1 ك²ة does not affect the dynamics of system Equation 19 it is included solely to adjust the zero-point energy the entire system. Substituting x' = 2/L, u = u/ L, and
C/T. into the above system, and then removing the primes, obtains:
OΦ
duu 02 22 -1)0,
(21a)
0q
220 7- - 1)0g
(21b)
0p
where the parameters and are defined by
To
EA
7 = L
and
a a= To
(22)
PA
For 4, one has p(p, g) = /(1 + p. Series approximations have been instrumental in analyzing nonlinear systems like the string model mentioned earlier; approximations up third or fourth order are frequently used [39]. The function Φ g) can be approximated with a variety of orders p and q:
1 ---
Φ 2
$ 2 2
+4
2---50 +²ク-0² 5"
In this paper, we consider the following approximation to ((p. 9), similar to 44, but lacking one of the fourth-order terms, following Bilbao
10.
1
2
P4
5"
2
Under this choice $7. the system Equation 21 reduces to
Ou 2= 0 (2 1)0, (1² 2pq)
(23a)
dec - -
110_ (q²)
(23b)
which can be augmented by the stiffinss and the damping term become Equation 3 The nonlinear damped stiff string system as Equation iss very similar to the work by Bank and Sujbert [2] but with the damping coefficients given as the same values for the longitudinal and transverse directions.
also noteworthy that there are many approaches to modeling the nonlinear string vibration, Frrchhoff-Carrie-ste
other than the aforementioned Transformation Method (FTM), which a powerful method for modeling the oscillation of acoustic systems in terms of transfer functions. Interested readers applications from (63H66). Those with background in neural network frameworks may also find reading [29] particularly engaging, as Schlecht et al. [29] points out the connection between the underlying mathematical ideas in the FTMs and the FNOs.
one. A good example is the Functional
advised to read its derivation and
C Damping coefficient
The damping coefficients and σ₁ are typically determined experimentally, especially those for nonlinear systems. However, can be somewhat difficult set the values without any prior knowledge of the observations. In this regard, the practical settings for the damping coefficients can approximately derived from the decay times, by resorting to the values for linear systems. The linear string damping coefficients do and are derived from the (frequency-dependeen) T60
values: where f(1) and f(2) denotes the two distinct frequencies. and (() and ((3) denotes the corresponding decay times. In this case,
Oo = 610g(10)
51 (2)
&
log
{1 32
=
ε3 E
(2)
(24)
T60
T60
T66
T60
where & = - 14 + 4x² x (2TT ( for i = 1,2. For this model, as the energy loss increases
monotonically with frequeecy, must choose (1)
Too when (11
(2)
(2)
TTo >
fT6o
17

---

Intial Condition
Tension, Time
(Plucked String?
Slifness Damping (T60)
Paremeter Encoder
Noise Decocier
Fealure
Estimator
Harmonic
Plich
Harmonic Amplitude
Integer Multiples
FM
Figure 9:
The DDSPish is designed in a similar way to DMSP,
D Baselines
In this section, we present more details of the baselines. All models, including DMSP, are trained using RAdam optimizer, with Noam learning rate scheduler with peak learning rate of 10 reaching at 1000 number of warmup steps. Figure 9 summarizes the architectures of DDSPish models. The only architectural difference between DDSPish and DDSPish-XEM is the existence the FM module. DDSPish-we is harmonic as the mode frequencies are not modulated after its initialization by the integer multiples of fo. while DDSPish is inharmonic FM trained to modulate the harmonic pitch skeleton to match the inharmonic mode frequencies.
For the Modal synthesis, we compute the solution [Equation with the allowed values of pas Mn, Ct, and C2 obtained using the Levenberg-Marquatte algorithm. We compute the modes up to the 100th order with the double-preed prebls-pr floating-point arithmetics. The modes are then post-processed to be cut under the Nyguist limit. Subsequently, all models are trained using 40 numbers of modes. We use 65 number bands used for the filtered noise.
Table 5: PDE parameter sampling range
Min.
Max.
Unit
fo 98.00
440.0
Hz
0.01y
0.03%
Q
1
(1)
10
25
sec
(2)
(T60
sec
TGO
1100
1200
Hz
(2)
T60
100
T60
1000
Hz
Po
0.001
0.02
Pr
0.1
0.5
E Datasets
As described in subsectiion 4.1 this paper utilizes the nonlinear string simulator, StringFDTD-Toree presented by Lee et al. [47]. For the simulation, PDE parameters are uniformly handom-sample within the moderate parameter ranges. Table summarizes the infimum and the supremum values for each uniform distribution of the PDE parameter. For a given random-sampled parameter set, the simulator outputs the transverse and the longitudinal solutions (u and Ç, resp.) while this work considers u only. Yet, it worth mentioning that the adopted u. is different from that of the one- dimensional string since and are coupled as evident in [Equation The obtained is defined over a spatio-temporal grid, where the spatial resolutions are carefully chosen to mitigate the numerical stability criteria and the numerical dispersions while keeping the temporal resolution by the prefixed audio sampling rate. As the grid spacing is fixed by these values, the raw simulation data are consisted of diverse grid sizes depending on fo, and a, making difficult batchify for training. For this reason, we spatially upsample the data to fixed spatial grid size of 256, using bivariate spline approximation over rectangular
mesh up to the 5th order degree.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We reflect the paper's contributions and scope to the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
We included the assumptions and proofs in the manuscript.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We tried our best to fully disclose all the information needed to reproduce the results, and willing to supplement any additional information, if needed.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We provide code and data attached in the link at the footnote.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee 7/nips.com htte:/ubbl///liiiee////eeeuuuisl ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the experimental results using appropriate statistics of the measured
scores.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computational resources for the dataset construction and the computation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in this paper conform with the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research proposes neural network for the physical simulation of musical instrument for scientific computing.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We tried our best to include as many proper credits for the original owners as possible by citing the original works. We will include the missing credits if any.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We documented the code and its usage properly, and will maintain the docu- mentation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "3LEDk2hDba", "reviewer": "Reviewer_gawP", "summary": "A computational framework that can approximate the motion of nonlinear strings is proposed. The implementation is differentiable, so one can train then neural nets in the framework as usual.", "questions": "I don't have major questions.", "limitations": "Limitations are simply stated in the conclusion section, which sound reasonable.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is written nicely such that I could follow the basic discussions even if I am not familiar with the topics of speech and audio. It seems to include all the information required in this kind of paper, such as technical background, model description, loss description, and experiments.\n- The empirical performance of the method is clearly superior to the reasonable baselines.", "weaknesses": "I have only minor comments as follows. Basically no need to respond to them.\n\n- In Table 1, the computational complexity part does not consider the computation needed for learning unknown parameters. The same kind of complexity analysis may be difficult for that part, but at least, the fact that such a training procedure happens for some of the methods should be noted somewhere close.\n\n- Lines 137 and 138: there seems to be confusion about the definition of the initial condition and/or the operator $\\mathcal{S}$. First, $u_0 \\in \\mathcal{U}$ sounds strange if $\\mathcal{U}$ is the space of functions $\\Omega \\times [0,\\infty) \\to \\mathbb{R}$ because $u_0$ does not take the time $\\in [0,\\infty)$ as the argument. Second, the domain of $\\mathcal{S}$ is the product of $\\mathcal{P}$ and the space of initial conditions, isn't it?\n\n- The role of the noise decoder is unclear. Is it important in the given experiments where the data are purely from the simulator?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer gawP for the constructive review. We also thank you for appreciating the novelties made by the differentiable string motion synthesis. Below are the responses to some of your comments. Each item in Weakness is labeled with a number following the W (from the top, W1, W2, ...)\n\n- **W1 (Specifying that the training procedure happens)**\n   - The authors agree that it's a good idea to clarify that differentiable methods need to be trained in advance. We will mention this in the caption, as we are concerned that labeling this as one of the columns in the table could potentially mislead some readers with the “Pre-computation” in the table, which is to apply the least-squares method for fitting the modes to each specific initial condition (IC). As reviewers recognize, there are differences between pre-computation and training:\n|  | Training | Pre-computation |\n| --- | ---: | ---: |\n| Performed before inference | &check; | &check; |\n| Performed every time for new ICs | &cross; | &check; |\n| Performed every time for new materials | &cross; | &check; |\n| Typical wall-time required | 0 days | 00 secs |\n\n      In the manuscript, Table 1 only refers to the inference scenario, and we believe that it is difficult to make a rigorous comparison of the computational complexity for training along the same lines. However, we agree with the reviewer that it would be nice to include the fact that neural networks require parameter optimization for inference. Following the reviewer’s suggestion, we included this in the caption of the table, albeit briefly.\n\n\n- **W2 (Problem statement)**\n   - Please see the global response (`Author Rebuttal by Authors`).\n\n- **W3 (Role of noise decoder)**\n   - The role of the noise decoder is not dominant, but it does help model the simulated nonlinear strings. As the nonlinear strings considered herein are modeled as the coupled system between the transverse and the longitudinal vibrations, the motion along the longitudinal axis contains strong near-noisy timbre along with (in)harmonic pitch skeletons (please see Figure 8.10 in [1]). This coupling leads to the appearance of timbres that can be approximated as filtered noise, especially in the transient region of the highly nonlinear strings with large pluck amplitudes. The noise decoder was designed to model such characteristics.\n\n    [1] Bilbao, S. (2009). *Numerical sound synthesis: finite difference schemes and simulation in musical acoustics*. John Wiley & Sons.\n\nPer the reviewer's advice, we will add them to the camera-ready version. We plan to add each of the points mentioned in W1, W2, and W3 to the caption of Table 1, Section 3.1 of the main text, and the Appendix. Again, we thank the reviewer for the constructive advice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1LwDMrMtK", "reviewer": "Reviewer_ygxN", "summary": "This paper proposes Differentiable Modal Synthesis for Physical Modeling (DMSP), which is a neural-model-based method to predict the vibration of nonlinear strings. DMSP takes the physical parameters for the differentiable equation as the input, predicts the mode frequencies and AM/FM effects, and finally outputs the waveform of vibration.", "questions": "Please see ‘weaknesses’.", "limitations": "Limitations are minimally discussed in Section 5. It would be nice to extend the discussion to include the failure of the mode predictor, the computational complexity of DMSP-N, etc.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "S1: This paper is among the first attempts to study neural audio synthesis using physical properties, which makes a meaningful contribution in this sub-area.\n\nS2: Section 2 provides some background which is helpful for understanding the context of the proposed method.", "weaknesses": "W1: Some method details are not introduced with sufficient clarity. In the loss section, the detailed mathematical forms for pitch loss and mode frequency loss are not explained. Is the pitch loss L2 loss, L1 loss, or in other forms? What type of regularization is used for the mode frequency loss? Also, since there are typically many mode frequencies for a linear/nonlinear system, it is unclear how many mode frequencies the mode estimator predicts. Is it a fixed number or a variable number? If it is a variable number, how did the neural model output a variable number of mode frequencies?\n\nW2: As the authors have also discussed, the mode predictor turns out to be not functioning appropriately based on the results in Table 2 (the big gap between DMSP and DMSP-N). Since the mode predictor is a major component of the proposed system, the design of the proposed pipeline is not well-justified.\n\nW3: It is generally unclear what advantages the proposed physics-driven sound synthesis paradigm has over neural-based generative models, such as AudioLM or diffusion models. The latter has been shown to synthesize high-quality sounds, music, and speech. Does the proposed paradigm have higher quality, lower computational overhead, greater generalizability, or better in other aspects than the latter paradigm?\n\nW4: (Minor) There are many empty references, e.g., line 135, line 229, and wrong references, e.g. Table 1 is line 243 which should probably be Table 2. Also, in line 243, ‘the ablation of the mode information is studied..’ Shouldn’t this table be the main result? Calling it an ablation result rather than the main result may cause some confusion, especially when Table 3 shows another ablation study result.\n\nW5: (Minor) The problem statement in 3.1 is a bit confusing to me. For example, it is unclear to me why the initial condition $u_0$, is an element of $\\mathcal{U}$. $u_0$ is a function on $\\Omega \\times \\{0\\}$, whereas $\\mathcal{U}$ is a set of $(x, t)$ pairs. Also, in line 138, it seems that the mapping $\\mathcal{S}$ is not only a function of $\\mathcal{P}$, but also a function of initial conditions. Why is the latter omitted?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer ygxN’s effort in reviewing our paper. Below are the responses to your concerns.\n\n- **W1 (Clarity in methodological details)**\n    - We clarify the mathematical definition of loss based on the reviewer's comments. We used $\\ell^1$ distance as $\\mathcal{L}_{f_0} = \\|\\hat{f_0} − f_0\\|_1$ for the metric between all mode frequencies; however, we found no significant deviation in training results according to the $p$ value of the $\\ell^p$-norms.\n    - As specified in Appendix D, we used 40 numbers of modes considering the fundamental frequency and the Nyquist limit by the temporal sampling frequency. This mode count was determined to be an integer value such that an integer multiple of the maximum fundamental frequency in the dataset is less than or equal to the Nyquist frequency, and as the reviewer notes, a more detailed synthesis would be possible if this mode count could be assigned dynamically according to $f_0, \\kappa$, and $\\alpha$.\n\n- **W2 (Role of the mode predictor)**\n    - With all due respect, we disagree with the reviewer’s point for two reasons:\n        - The **Modal synthesis is pretty much just a solution in itself for linear strings** (the difference between Modal and FDTD comes only from the frequency-dependent damping term, compare Eqn. 1 and 3 for $\\alpha=1$), so it is perfectly natural for other models, including DMSP, to lag behind Modal, as the training data does not have $\\alpha$ exactly $1$ like the Linear ($\\alpha=1$) test set. Thus, the discrepancy in DMSP's Linear string test result compared to Modal's **does not imply that DMSP’s mode estimator doesn't work at all.**\n        - As the title of this paper suggests, our main interest is on planar (thus nonlinear; see Section 2.1) strings. As the Nonlinear string test result suggests, **the mode predictor is not a major component**: DMSP still outperforms the rest of the baselines and ranks second, suggesting that even with errors in the mode estimator, synthesis results can be better.\n\n        Our statement in L255 means that \"errors can occur in estimating the mode through a neural network\", and nowhere in the text states that the mode estimator is the main module. However, when it comes to the wording of this sentence, we agree that it could be potentially misleading to readers, and we will refine the sentence to be clearer about what we're claiming.\n\n    - The following additional experimental results further demonstrate how trivial the error in the mode predictor is in the overall proposal.\n        |  | Linear | (\\\\(\\alpha=1\\\\)) |||| Nonlinear |  (\\\\(\\alpha>1\\\\)) |||\n        | --- | ---: | ---: | ---: | ---: | --- | ---: | ---: | ---: | ---: |\n        |  | **SI-SDR**  | **SDR** | **MSS**  | **Pitch** || **SI-SDR**  | **SDR**  | **MSS**  | **Pitch**  |\n        | Modal | **–3.191** | **0.681** | 18.449 | ***0.420*** || –16.611 | –1.900 | 17.254 | 2.316 |\n        | DDSPish | –39.478 |  –2.598 | **11.047** | 5.518 || –25.951 | –2.102 | 9.745 | 3.306 |\n        | DDSPish-woFM | –46.609 | –2.257 | ***10.911*** | 11.304 || –46.858 | –2.272 | 10.299 | 14.013 |\n        | **DMSP-N** | ***–2.844*** | ***1.496*** | 12.525 | **0.792** || ***15.670*** | ***16.455*** | ***4.772*** | **1.027** |\n        | **DMSP** | –22.298 | –2.000 | 12.504 | 1.717 || **–10.315** | **0.221** | **5.656** | ***1.437*** |\n\n        For this additional experiment, we increased the training data and the training time (about a week using a single 2080), while keeping all other network structural details the same. For the Linear results, DMSP still lags behind DMSP-N and Modal, suffering from its disadvantage in mode estimation accuracy, but clearly outperforms the baselines in nonlinear strings. This emphasizes that the error of the mode estimator is not a critical part of the overall pipeline, but rather the decoder part.\n\n- **W3 (Comparison with generative models)**\n    - While we fully understand the reviewer's curiosity, we believe that providing a formal analysis of this point in the text would detract from the main argument:\n        1. As we believe the reviewer also agrees with (as noted in Strength S1 of the review), **there is no prior work on generative models based on neural networks (such as AudioLM or diffusion) for synthesizing string motion**, i.e., the time-dependent displacement of a string represented by the solution of a PDE, as in this study.\n        2. The main \bpoint of the paper is **'making one of the physical modeling methods (modal synthesis) differentiable facilitates efficient and effective nonlinear string synthesis'**, and it is considered somewhat out of context to add claims such as 'the proposed model structure and training method is even better/worse than those trained with the diffusion framework and/or LLM training method'.\n\n      Nevertheless, we tried various architectures (such as Transformer and WaveNet) and also trained conditional generation on top of a diffusion framework (like DiffWave [1] or Music Spectrogram Diffusion [2]). Please see the PDF file attached to the global response for a comparison under different architectures. Most attempts were prone to fail in motion synthesis, i.e., modeling the physical correlation of displacement with position, and DMSP performs well because it directly utilizes mode information to represent the physical correlation.\n\n        [1] Kong, Z., Ping, W., Huang, J., Zhao, K., & Catanzaro, B. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In *International Conference on Learning Representations*, 2021.\n\n        [2] Hawthorne, C., Simon, I., Roberts, A., Zeghidour, N., Gardner, J., Manilow, E., & Engel, J. (2022, December). Multi-instrument Music Synthesis with Spectrogram Diffusion. In *ISMIR 2022 Hybrid Conference*.\n\n- **W4 (Typos and missing references)**\n    - We thank the reviewers for pointing this out. Please see the global response (`Author Rebuttal by Authors`).\n\n- **W5 (Problem statement)**\n    - Please see the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLdY1x2e1G", "reviewer": "Reviewer_Hh9k", "summary": "The paper presents a differentiable model that can synthesize musical string sound and simulate motion based on physical properties. The method uses finite-difference time-domain (FDTD) solver to obtain numerical solutions and take them as ground truth. Then a differentiable pipeline with neural network components is used to map physics properties to output waveform. Experimental results show that it achieves better performance than Modal synthesis and DDSPish baselines.", "questions": "See questions in weakness", "limitations": "Authors mentioned limitations in the main text.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- To my best knowledge, it is the first differentiable method that can generate the musical string sound and motion from physics properties.\n- The proposed method is more efficient and accurate than baselines.", "weaknesses": "- The proposed method relies FDTD solutions as ground truth and I am wondering what will be the gap between the simulated solutions versus real-world audio data. \n- Following the previous question, there might not be too many real-world data available for learning and usually material properties are unavailable. Is it possible to infer these properties through the differentiable pipeline?\n- It needs ablation studies for using two MLPs for modulation layers, as well as a MLP for mode estimator.\n- There are references typos in L135 and L229.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Hh9k for the extensive review. Below are the responses to your concerns. Each item in Weakness is labeled with a number following the W (from the top, W1, W2, ...)\n\n\n- **W1 (The gap between the simulation and the real-world audio data)**\n    - We summarized some of the main differences between the simulated data (using StringFDTD-Torch [1] or DMSP) and the real-world audio data as below.\n        \n        | | Simulated |  Recorded |\n        | --- | :---: | :---: |\n        | Can obtain sound | &check; | &check; |\n        | Can obtain displacement (i.e. movement) | &check; | &cross; |\n        | Free from measurement errors | &check; | &cross; |\n        | Free from modeling errors | &cross; | &check; |\n        | Close to the 'sounds' of everyday life* | &cross; | &check; | \n        #### *This is not true for all FDTD simulations, but it is true for the simulations covered in this manuscript.\n\n        The first four rows illustrate the main differences in system modeling.\n\n        - **Simulated data** has the advantage that the displacements of the strings are directly found as solutions of the PDE, giving **access to the motion information for all positions** and that there is **little measurement error**; however, since the string system is represented by a parametric PDE, this does not prevent the modeling itself from introducing errors, but there have been many validation studies showing that these **modeling errors are small enough** [2-3].\n        - On the other hand, **recorded audio** data can be said to have **no modeling error** because it drives the actual string as it is, but it is characterized by a large loss of information in that it uses a specific receiver sensor to record audio data. \bThe audio data records the vibration of the measurement equipment (e.g. microphone membrane) transmitted from the vibration of the string to the receiver at a specific location, through the medium of the measurement environment. So the **displacement of the string is unknown**, and **measurement errors** (such as microphone coloration or room reverberations) are inevitably introduced.\n\n        The last row of the table is a slightly different context to this systematic difference and considers its value as a musical instrument.\n\n        - As described in the manuscript, and as the systematic difference suggests, simulated audio is significantly different from real recorded audio, since we are in effect “listening to the displacement” picked up at a particular location on the string. The system in which the vibration of a string propagates through the air and is recorded at the receiver, as we hear in our daily lives, is another active area of research [4]. Based on these studies and ours, we believe that in the future, **we will be able to further bridge the gap between simulated data and actual recorded data**.\n        \n        [1] Lee, J. W., Choi, M. J., & Lee, K. (2024, April). String Sound Synthesizer On GPU-Accelerated Finite Difference Scheme. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1491-1495). IEEE.\n\n        [2] Bensa, J., Bilbao, S., Kronland-Martinet, R., & Smith III, J. O. (2003). The simulation of piano string vibration: From physical models to finite difference schemes and digital waveguides. *The Journal of the Acoustical Society of America*, *114*(2), 1095-1107.\n        \n        [3] Ducceschi, M., & Bilbao, S. (2022). Simulation of the geometrically exact nonlinear string via energy quadratisation. *Journal of Sound and Vibration*, *534*, 117021.\n        \n        [4] Bilbao, S., & Ahrens, J. (2020). Modeling continuous source distributions in wave-based virtual acoustics. *The Journal of the Acoustical Society of America*, *148*(6), 3951-3962.\n        \n- **W2 (Possibility of inferring the material properties through the proposed method)**\n    - In response to this question, which holds good insights for further research, we share the results of an informal experiment. From what we have tried, we believe that it is not impossible, but requires some tricks and a well-controlled experimental setup to get meaningful conclusions. For example, the problem of estimating string’s material properties and initial conditions from single-channel audio recordings is ill-posed and it is well-known [5] that there can be multiple solutions to similar problems. However, with better DMSP models (as mentioned in the answer to the previous question) and tricks to better address ill-posedness for properties, we are optimistic about solving this problem (e.g., inferring the material properties and the initial conditions from the sound.)\n        \n        [5] Kac, M. (1966). Can one hear the shape of a drum?. *The american mathematical monthly*, *73*(4P2), 1-23.\n        \n- **W3 (Ablations with the number of MLP layers)**\n    - In response to a reviewer's question, we found that the difference in performance based on the number of MLP layers is insignificant, which is why we did not report it separately in the manuscript. This can be taken similarly to what we said in the text about using GRUs instead of MLPs, which is that **there is a small difference in performance, but not enough to change the ranking between models**. Modifying the details for each module did not have a significant impact on the performance change, which is consistent with conventional knowledge: Stacking more layers than fewer generally performs better, but the performance gains taper off, so we chose a modest number of layers, taking into account GPU memory and batch size. The most important factor for model performance is how the mode information is utilized, \band \bwe found the decoder structure that modulates a sinusoidal oscillator through AM and FM performs best for motion synthesis.\n\n- **W4 (Typos and missing references)**\n    - Thank you so much for pointing this out. Please see the global response (`Author Rebuttal by Authors`).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for taking the time to review the paper and for their efforts to improve the quality of the manuscript with their constructive comments. We responded to each reviewer's comments and concerns individually, and the responses below address common points made by reviewers.

- **Missing references, typos, and misleading expressions are revised as follows.**
    - L135: "as specified in Equation 4,"
    - L229: "linear wave solution as in Equation 1,"
    - L243: "The efficacy of DMSP is studied as shown in Table 2."
- **Problem statement is revised as follows.**
    - We assume that the solution $u : \Omega × [0, \infty) \to \mathbb{R}$ resides within the Banach space $\mathcal{U}$.
       For a given PDE parameter $\rho\in\mathcal{P}$ and initial condition $u_0\in\mathcal{U}\_0\subset\mathcal{U}$ with $u\_0:\Omega\times\{0\}\to\mathbb{R}$, let $\mathcal{S}:\mathcal{P}\times\mathcal{U}\_0\to\mathcal{U}$ denote a nonlinear map, specifically the FDTD numerical solver tailored to the context of this study.

        Assume that we are provided with observations $\\{\rho^{(i)}, u\_0^{(i)}, u^{(i)}\\}^N_{i=1}$, where $\rho^{(i)}$ and $u\_0^{(i)}$ are i.i.d. samples drawn from a probability measure supported on $\mathcal{P}$ and $\mathcal{U}\_{0}$ respectively, and $u^{(i)} = \mathcal{S}(\rho^{(i)},u\_0^{(i)})$ potentially contains noise.
        Our goal is to construct an approximation of $\mathcal{S}$ denoted as $\mathcal{S}\_\theta : \mathcal{P}\times\mathcal{U}\_0 \to \mathcal{U}$, and select parameters $\theta^*\in\mathbb{R}^{N\_\theta}$ such that

        $$
        \min_{\theta}\\mathbb{E}\_{\rho\sim\mu\_{\mathrm{pa}},u\_0\sim\mu\_{\mathrm{ic}}}\left\\|\mathcal{S}(\rho,u\_0) - \mathcal{S}\_\theta(\rho,u_0)\right\\|\_{\mathcal{U}}\approx\min\_\theta\frac{1}{N}\sum\_{i=0}^N\left\\|\mathcal{S}(\rho^{(i)},u\_0^{(i)}) - \mathcal{S}\_\theta(\rho^{(i)},u\_0^{(i)})\right\\|\_{\mathcal{U}}
        $$
    
        where $\rho^{(i)}\sim\mu\_{\mathrm{pa}}$ and $u_0^{(i)}\sim\mu\_{\mathrm{ic}}$. Leveraging $\mathcal{S}\_\theta$, one can compute the solution $\hat{u} = \mathcal{S}\_\theta(\rho,u\_0)$ corresponding to a new parameter $\rho\in\mathcal{P}$ and a new initial condition $u\_0\in\mathcal{U}\_0$. By specifying values for $x$ and $t$, one can then either synthesize the sound of the string picked-up (also referred to as read-out) at a specific location $x_0$ as $\hat{u}(x\_0, t)$, or simulate the motion of the string by concatenating $\hat{u}(x, t)$ across all $x \in\Omega$.


In addition to this answer, the **attached PDF file includes** the following:

- a table of **synthesis results for the improved model** from additional experiments,
- a table of **ablation study for the improved model**,
- a comparison table of training results for **various neural network architectures** such as Transformer and WaveNet,
- a **scatter plot of the improved scores**, and
- a figure showing the **motion synthesis results** over time for different pluck positions.

Meta-review: The paper introduces a novel model for simulating the spatio-temporal motion of nonlinear strings by integrating modal synthesis and spectral modeling within a neural network framework. 

It is highlighted as the first differentiable method capable of generating musical string sound and motion from physical properties. The empirical performance of the proposed method is noted to be superior to reasonable baselines. The reviewers appreciate that the paper is well-written and easy to follow, even for those not familiar with speech and audio topics. It includes all necessary information such as technical background, model description, loss description, and experiments.

Most of the reviewers' concerns have been well addressed during the rebuttal period.
Decision: Accept (poster)