# Melodic Constraint Satisfaction with a Generative Adversarial Network

Tsubasa Tanaka

► To cite this version:

Tsubasa Tanaka. Melodic Constraint Satisfaction with a Generative Adversarial Network. Proceedings of International Computer Music Conference 2023, Oct 2023, Shenzen, China. hal-04560730

HAL Id: hal-04560730

https://hal.science/hal-04560730

Submitted on 29 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Melodic Constraint Satisfaction with a Generative Adversarial Network

Tsubasa Tanaka

► To cite this version:

Tsubasa Tanaka. Melodic Constraint Satisfaction with a Generative Adversarial Network. Proceedings of International Computer Music Conference 2023, Oct 2023, Shenzen, China. hal-04560730

HAL Id: hal-04560730

https://hal.science/hal-04560730v1

Submitted on 29 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Melodic Constraint Satisfaction with a Generative Adversarial Network

Tsubasa Tanaka

Tokyo University of the Arts

tanaka.tsubasa@ms.geidai.ac.jp

## ABSTRACT

Generative Adversarial Networks (GANs) are typically used in order to generate instances that resemble those in a data set. However, this study generalizes GAN so that it can be used as a method of constraint satisfaction. If GANs get the capacity to satisfy explicit rules that do not necessarily exist in the training data set, this may improve the controllability and trustfulness of generative AI. An experiment of a simple melodic generation suggests that the GAN can learn constraints from synthetic data sets for respective constraints and generate solutions that simultaneously satisfy all of the constraints.

## 1. INTRODUCTION

Deep generative models such as GANs, VAEs, and diffusion models have been extensively applied in artistic fields such as painting, photography, music, and poetry.

Among the deep generative models, GAN is a unique neural network structure that combines two different networks called a generator and a discriminator. While the discriminator tries to distinguish real and fake (generated) samples, the generator tries to convince the discriminator that the generated fake samples are real. In this way, both networks compete with and enhance each other.

This learning process of GANs is formulated as the following two-player minimax game [1]:

$$ \min_G \max_D \left\{ E_x[\log(D(\mathbf{x}))] + E_z[\log(1 - D(G(\mathbf{z})))] \right\} \quad (1) $$

, where $\mathbf{x}$ is sampled from the data distribution and $\mathbf{z}$ is sampled from random noise.

After the emergence of the original GAN, many variants of GAN have been proposed. DCGAN (Deep Convolutional GAN) adopts a convolutional neural network (CNN) to generate clearer images [2]. Conditional GAN enables conditional generation by additional input information [3]. CycleGAN combines two GANs for image translation [2]. StyleGAN enables style manipulation using style vectors [4]. There are applications of GAN in the field of music generation such as [5].

In this study, we focus on GANs' mechanism of the discriminator and generalize it so that GANs can treat a wider range of tasks, especially solving constraint satisfaction problems.

## 2. PROPOSED METHOD

### 2.1 Generalizing GAN

The original GAN's objective is to make indistinguishable imitations from real data. In other words, the objective is to generate solutions that satisfy a criterion of genuineness that the discriminator has learned.

In analogy, we can imagine a generalization of GAN that have multiple criteria and target degrees of satisfying these criteria. In this case, we can introduce other criteria that are desirable to be realized instead of only genuineness. This is important because sticking to genuineness alone makes it difficult to deal with something more creative than imitating what already exists.

Therefore, the author proposes the following generalized formulation of GAN: the discriminator $D$ learns to minimize the following loss function

$$ L_d(\theta_d) := E_x[\|R_x - D(\mathbf{x})\|^2] + E_z[\|R_{G(\mathbf{z})} - D(G(\mathbf{z}))\|^2] \quad (2) $$

, and the generator $G$ learns to minimize the following loss function

$$ L_g(\theta_g) := E_z[\|T - D(G(\mathbf{z}))\|^2], \quad (3) $$

where $R_x \in [0, 1]^r$ denotes a vector that expresses how much $\mathbf{x}$ satisfies $r$ criteria and $T \in [0, 1]^r$ denotes a constant vector that indicates the target degrees of satisfaction of the $r$ criteria. $\theta_d$ and $\theta_g$ are the parameters of $D$ and $G$ respectively.

Equation (2) means that $D$ learns to accurately evaluate data $X$ and generated samples $G(\mathbf{z})$ concerning the degrees of satisfaction of the criteria. Equation (3) means that $G$ learns to generate samples that realize $T$, the target degrees of satisfaction of the criteria (rules or constraints), on the assumption that $D$ can accurately output $R_{G(\mathbf{z})}$.

### 2.2 Implementation

The two minimization processes are implemented as iterative alternations of updating the parameters of $D$ (Equation (2)) and those of $G$ (Equation (3)) with backpropagation, as in the learning process of ordinary GANs.

## 3. APPLICATION IN MELODIC GENERATION

To verify the capacity of the proposed method, the author conducted an experiment on simple music generation with melodic constraints. For simplicity, this experiment only treats a minimum number of rules taken from music theory, although these are not sufficient from the viewpoint of musical quality. This Section explains these constraints and how there are formalized in the experiment, which is described in the next section.

Copyright: ©2023 Tsubasa Tanaka et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
# Melodic Constraint Satisfaction with a Generative Adversarial Network

Tsubasa Tanaka

Tokyo University of the Arts

tanaka.tsubasa@ms.geidai.ac.jp

## ABSTRACT

Generative Adversarial Networks (GANs) are typically used in order to generate instances that resemble those in a data set. However, this study generalizes GAN so that it can be used as a method of constraint satisfaction. If GANs get the capacity to satisfy explicit rules that do not necessarily exist in the training data set, this may improve the controllability and trustfulness of generative AI. An experiment of a simple melodic generation suggests that the GAN can learn constraints from synthetic data sets for respective constraints and generate solutions that simultaneously satisfy all of the constraints.

## 1. INTRODUCTION

Deep generative models such as GANs, VAEs, and diffusion models have been extensively applied in artistic fields such as painting, photography, music, and poetry.

Among the deep generative models, GAN is a unique neural network structure that combines two different networks called a generator and a discriminator. While the discriminator tries to distinguish real and fake (generated) samples, the generator tries to convince the discriminator that the generated fake samples are real. In this way, both networks compete with and enhance each other.

This learning process of GANs is formulated as the following two-player minimax game [1]:

$$ \min_G \max_D \left\{ E_x[\log(D(x))] + E_z[\log(1 - D(G(z)))] \right\} \quad (1) $$

, where $x$ is sampled from the data distribution and $z$ is sampled from random noise.

After the emergence of the original GAN, many variants of GAN have been proposed. DCGAN (Deep Convolutional GAN) adopts a convolutional neural network (CNN) to generate clearer images [2]. Conditional GAN enables conditional generation by additional input information [3]. CycleGAN combines two GANs for image translation [2]. StyleGAN enables style manipulation using style vectors [4]. There are applications of GAN in the field of music generation such as [5].

In this study, we focus on GANs' mechanism of the discriminator and generalize it so that GANs can treat a wider range of tasks, especially solving constraint satisfaction problems.

## 2. PROPOSED METHOD

### 2.1 Generalizing GAN

The original GAN's objective is to make indistinguishable imitations from real data. In other words, the objective is to generate solutions that satisfy a criterion of genuineness that the discriminator has learned.

In analogy, we can imagine a generalization of GAN that have multiple criteria and target degrees of satisfying these criteria. In this case, we can introduce other criteria that are desirable to be realized instead of only genuineness. This is important because sticking to genuineness alone makes it difficult to deal with something more creative than imitating what already exists.

Therefore, the author proposes the following generalized formulation of GAN: the discriminator $D$ learns to minimize the following loss function

$$ L_d(\theta_d) := E_x[\|R_x - D(x)\|^2] + E_z[\|R_{G(z)} - D(G(z))\|^2] \quad (2) $$

, and the generator $G$ learns to minimize the following loss function

$$ L_g(\theta_g) := E_z[\|T - D(G(z))\|^2], \quad (3) $$

where $R_x \in [0, 1]^r$ denotes a vector that expresses how much $x$ satisfies $r$ criteria and $T \in [0, 1]^r$ denotes a constant vector that indicates the target degrees of satisfaction of the $r$ criteria. $\theta_d$ and $\theta_g$ are the parameters of $D$ and $G$ respectively.

Equation (2) means that $D$ learns to accurately evaluate data $X$ and generated samples $G(z)$ concerning the degrees of satisfaction of the criteria. Equation (3) means that $G$ learns to generate samples that realize $T$, the target degrees of satisfaction of the criteria (rules or constraints), on the assumption that $D$ can accurately output $R_{G(z)}$.

### 2.2 Implementation

The two minimization processes are implemented as iterative alternations of updating the parameters of $D$ (Equation (2)) and those of $G$ (Equation (3)) with backpropagation, as in the learning process of ordinary GANs.

## 3. APPLICATION IN MELODIC GENERATION

To verify the capacity of the proposed method, the author conducted an experiment on simple music generation with melodic constraints. For simplicity, this experiment only treats a minimum number of rules taken from music theory, although these are not sufficient from the viewpoint of musical quality. This Section explains these constraints and how there are formalized in the experiment, which is described in the next section.

Copyright: ©2023 Tsubasa Tanaka et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
## 3.1 Melodic Constraints

When creating melodies, there are rules that composers implicitly or explicitly obey. In counterpoint [6, 7, 8], which is a theory for composing Western polyphonic music, there are many explicit rules that melodies are recommended to satisfy or must satisfy. In the context of computer music research, counterpoint has been studied from various viewpoints including constraint programming [9, 10], stochastic models [11, 12], genetic algorithm [13], and integer programming [14].

From the rules of counterpoint, we adopt the following four rules in this experiment:

* Rule 1: basically, pitches are within a diatonic scale.

* Rule 2: specific melodic intervals should not be used.

* Rule 3: conjunct motion should be frequently used.

* Rule 4: turns should not be used too frequently.

## 3.2 Degrees of Constraint Satisfaction

To evaluate how much these rules are satisfied, we introduce a vector $R_x = (r_1, r_2, r_3, r_4) \in [0, 1]^4$, which denotes the degrees of satisfying four rules for a melody $x$. $r_1$ corresponds to the rate of pitches within a diatonic collection {$C, D, E, F, G, A, H$} for the melody. For example, if a melody $x$ only uses pitches within a diatonic scale, $r_1$ equals 1.0. $r_2$ corresponds to the rate of occurrences of melodic intervals recommended in counterpoint. In counterpoint, there are prohibited melodic intervals such as large leaps over an octave and dissonant intervals. Repetition of the same notes is also not recommended. Therefore, we set the recommended melodic intervals as $I = \{1, 2, 3, 4, 5, 7, 8, 9, 10, 12\}$ by semitone (both ascending and descending directions are possible). $r_3$ corresponds to the rate of conjunct motions between consecutive notes, namely the move of one semitone and that of two semitones. In counterpoint, conjunct motions are recommended to be used frequently. Therefore, the value $r_3$ should be large. $r_4$ corresponds to the rate of direction changes in the melody (from ascending to descending or vice versa). For example, if the melodic direction changes at every note, $r_4$ equals 1.0 (completely zigzag). In counterpoint, frequent changes of melodic direction are not recommended. This is because if there are too many direction changes, the unity of the melody is destroyed. Therefore, the value $r_4$ should be small.

Considering these four rules, we set the target rate $T$ as (1.0, 1.0, 0.66, 0.36). This means that melodies that $G$ generate must strictly respect a diatonic scale and the recommended set of melodic intervals. Also, melodic motions must be musically well-controlled (conjunct motions are frequent, and turns are not too frequent).

## 4. EXPERIMENT

This section reports the result of the experiment to confirm that the proposed method works. This experiment aims to evaluate the capacity of the proposed variant of GAN to generate melodies with target vector $T$.

The role of discriminator $D$ is to identify correct values of $R_x$. Hopefully, $D$ can identify $R_x$ for any input melody $x$ after learning.

On the other hand, the role of generator $G$ is to generate melodies whose $R_x$ equals $T$. In this way, this GAN is expected to generate melodies that satisfy all of the rules.

## 4.1 Preparation of Synthetic Data Sets

To make a neural net learn the four melodic rules, the author prepared four synthetic melody data sets for respective rules $S_i(i = 1, 2, 3, 4)$. The strategy is to make the neural net learn each rule independently. It is not difficult to generate random pitch sequences specifying only one of the values of $r_i(i = 1, 2, 3, 4)$. However, generating random pitch sequences specifying all of the four values is difficult because the constraints become strict. This is why we use four different data sets.

The difficulty is that four synthetic data sets for respective rules, $S_i(i = 1, 2, 3, 4)$, do not necessarily contain ideal solutions whose $R_x$ coincide with $T$. Therefore, solving the problem of extrapolation is necessary ($G$ is expected to generate ideal solutions even if there is no such solution in the training data).

The number of samples in each data set $S_i$ was set to 1300. These were generated by programming. When generating, the training data set $i$ was only controlled to have every possible degree of satisfaction of the rule $i$ in an equal frequency, and other rules than rule $i$ were not controlled for data set $i$. The number of notes was fixed at 13. For simplicity, the durations of all 13 notes were set to a whole note. In the data sets, there was no melody whose $R_x$ precisely coincides with $T$. The nearest sample to the target $T$ had the values $R_x = (0.92, 1.0, 0.58, 0.36)$.

Test data sets were also generated in a similar way. The number of melodies in a data set for rule $i$ was set to 65.

## 4.2 Neural Network Structure

The proposed variant of GAN was implemented as follows. The discriminator $D$ consists of four fully-connected layers. The first three layers have 256 nodes with Leaky ReLU activation function and dropout. The last layer has four nodes with sigmoid activation function. The four outputs correspond to the values of the four rules. The generator $G$ consists of three fully-connected layers. The first two layers have 624 nodes with batch normalization and ReLU activation function. The third layer has $13 \times 24 = 312$ nodes. The output of the third layer is reshaped into a $13 \times 24$ matrix and output through a softmax function so that the final output becomes a sequence of 13 probability distributions of 24 classes. Each probability distribution corresponds to one note, and 24 classes correspond to the pitch range of 24 semitones (two octaves). The input of $G$ is random noise of dimension 100. Adam optimizer with a learning rate of 0.01 is used.

When calculating $R_{G(z)}$, the output of $G$ is converted into a melody (a series of one-hot vectors) by argmax decisions. Then, this melody is checked, and $r_i$ is calculated. However, when calculating $D(G(z))$, $G(z)$ is directly passed to $D$ without converting to on-hot vectors.

## 4.3 Result

Figure 1 shows the transitions of losses in the learning process of the GAN. The losses of both $D$ (Equation (2)) and
## 3.1 Melodic Constraints

When creating melodies, there are rules that composers implicitly or explicitly obey. In counterpoint [6, 7, 8], which is a theory for composing Western polyphonic music, there are many explicit rules that melodies are recommended to satisfy or must satisfy. In the context of computer music research, counterpoint has been studied from various viewpoints including constraint programming [9, 10], stochastic models [11, 12], genetic algorithm [13], and integer programming [14].

From the rules of counterpoint, we adopt the following four rules in this experiment:

* Rule 1: basically, pitches are within a diatonic scale.

* Rule 2: specific melodic intervals should not be used.

* Rule 3: conjunct motion should be frequently used.

* Rule 4: turns should not be used too frequently.

## 3.2 Degrees of Constraint Satisfaction

To evaluate how much these rules are satisfied, we introduce a vector $R_x = (r_1, r_2, r_3, r_4) \in [0, 1]^4$, which denotes the degrees of satisfying four rules for a melody $x$. $r_1$ corresponds to the rate of pitches within a diatonic collection {$C, D, E, F, G, A, H$} for the melody. For example, if a melody $x$ only uses pitches within a diatonic scale, $r_1$ equals 1.0. $r_2$ corresponds to the rate of occurrences of melodic intervals recommended in counterpoint. In counterpoint, there are prohibited melodic intervals such as large leaps over an octave and dissonant intervals. Repetition of the same notes is also not recommended. Therefore, we set the recommended melodic intervals as $I = \{1, 2, 3, 4, 5, 7, 8, 9, 10, 12\}$ by semitone (both ascending and descending directions are possible). $r_3$ corresponds to the rate of conjunct motions between consecutive notes, namely the move of one semitone and that of two semitones. In counterpoint, conjunct motions are recommended to be used frequently. Therefore, the value $r_3$ should be large. $r_4$ corresponds to the rate of direction changes in the melody (from ascending to descending or vice versa). For example, if the melodic direction changes at every note, $r_4$ equals 1.0 (completely zigzag). In counterpoint, frequent changes of melodic direction are not recommended. This is because if there are too many direction changes, the unity of the melody is destroyed. Therefore, the value $r_4$ should be small.

Considering these four rules, we set the target rate $T$ as (1.0, 1.0, 0.66, 0.36). This means that melodies that $G$ generate must strictly respect a diatonic scale and the recommended set of melodic intervals. Also, melodic motions must be musically well-controlled (conjunct motions are frequent, and turns are not too frequent).

## 4. EXPERIMENT

This section reports the result of the experiment to confirm that the proposed method works. This experiment aims to evaluate the capacity of the proposed variant of GAN to generate melodies with target vector $T$.

The role of discriminator $D$ is to identify correct values of $R_x$. Hopefully, $D$ can identify $R_x$ for any input melody $x$ after learning.

On the other hand, the role of generator $G$ is to generate melodies whose $R_x$ equals $T$. In this way, this GAN is expected to generate melodies that satisfy all of the rules.

## 4.1 Preparation of Synthetic Data Sets

To make a neural net learn the four melodic rules, the author prepared four synthetic melody data sets for respective rules $S_i(i = 1, 2, 3, 4)$. The strategy is to make the neural net learn each rule independently. It is not difficult to generate random pitch sequences specifying only one of the values of $r_i(i = 1, 2, 3, 4)$. However, generating random pitch sequences specifying all of the four values is difficult because the constraints become strict. This is why we use four different data sets.

The difficulty is that four synthetic data sets for respective rules, $S_i(i = 1, 2, 3, 4)$, do not necessarily contain ideal solutions whose $R_x$ coincide with $T$. Therefore, solving the problem of extrapolation is necessary ($G$ is expected to generate ideal solutions even if there is no such solution in the training data).

The number of samples in each data set $S_i$ was set to 1300. These were generated by programming. When generating, the training data set $i$ was only controlled to have every possible degree of satisfaction of the rule $i$ in an equal frequency, and other rules than rule $i$ were not controlled for data set $i$. The number of notes was fixed at 13. For simplicity, the durations of all 13 notes were set to a whole note. In the data sets, there was no melody whose $R_x$ precisely coincides with $T$. The nearest sample to the target $T$ had the values $R_x = (0.92, 1.0, 0.58, 0.36)$.

Test data sets were also generated in a similar way. The number of melodies in a data set for rule $i$ was set to 65.

## 4.2 Neural Network Structure

The proposed variant of GAN was implemented as follows. The discriminator $D$ consists of four fully-connected layers. The first three layers have 256 nodes with Leaky ReLU activation function and dropout. The last layer has four nodes with sigmoid activation function. The four outputs correspond to the values of the four rules. The generator $G$ consists of three fully-connected layers. The first two layers have 624 nodes with batch normalization and ReLU activation function. The third layer has $13 \times 24 = 312$ nodes. The output of the third layer is reshaped into a $13 \times 24$ matrix and output through a softmax function so that the final output becomes a sequence of 13 probability distributions of 24 classes. Each probability distribution corresponds to one note, and 24 classes correspond to the pitch range of 24 semitones (two octaves). The input of $G$ is random noise of dimension 100. Adam optimizer with a learning rate of 0.01 is used.

When calculating $R_{G(z)}$, the output of $G$ is converted into a melody (a series of one-hot vectors) by argmax decisions. Then, this melody is checked, and $r_i$ is calculated. However, when calculating $D(G(z))$, $G(z)$ is directly passed to $D$ without converting to on-hot vectors.

## 4.3 Result

Figure 1 shows the transitions of losses in the learning process of the GAN. The losses of both $D$ (Equation (2)) and
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Transitions of the MSE in the learning process of *D* with both training data and generated data (Equation (2)), *D* with generated data (the second term of Equation (2)), *D* with test data, and *G* with generated data (Equation (3)).</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Figure 2.** The transitions of the averages of $|r_i - t_i|$ for respective $i$ ($i = 1, 2, 3, 4$). These indicate to what extent respective rules are satisfied.</figcaption></figure>

*G* (Equation (3)) converged to 0.0, which implies that discriminations of both training data (D) and generated samples (D generated) were successful and *G* have acquired the capacity of satisfying the constraints (Equation (3)).

The loss of discrimination of the test data (D test data) continuously decreases, and the mean loss (MSE) is below 10%

Figure 2 shows the learning processes of respective rules (the transitions of the averages of $|r_i - t_i|$ for respective $i$ ($i = 1, 2, 3, 4$). Figure 3 shows the transitions of averages of $r_i$ for each $i$. This Figure indicates that all of the four elements of $R_\omega$ converge to the target values of $T$. We can observe that $r_1$ converges more accurately than others, which is understandable. The satisfaction of Rule 1 is easier because whether a note in a melody is inside the scale or not is independent of other notes.

Figure 4 shows some generated melodies whose loss was almost 0.0 (optimal). We can observe that the contours of these samples are similar. This is probably because the generator lost its diversity through the learning process and focused on specific area of optimal solutions.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** The transitions of averages of $r_i$ for respective $i$ ($i = 1, 2, 3, 4$). These indicate that $R_\omega$ successfully converged to $T$ (the horizontal lines).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Some examples of generated melodies. These samples realize $R_\omega = T$ (= (1.0, 1.0, 0.66, 0.36)), which means the constraints are perfectly satisfied. We can observe that there is no note outside of a diatomic scale, that all melodic intervals are included in the recommended set of intervals, that there are many conjunct motions, and that there are not too many direction changes.</figcaption></figure>

## 4.4 Discussion

From the result of the experiment, we have confirmed that the proposed GAN method has a capacity for constraint satisfaction. More precisely, the discriminator *D* could learn each rule “separately” from each synthetic data set and could teach the generator *G* to obey multiple rules simultaneously.

Although the author has conducted a preliminary experiment with another method based on a conditional VAE of which the target vector *T* is used as the conditional input, it was not successful. An important difference between this method and the proposed method is whether there is a mechanism of discriminator or not. In the case of conditional VAE, although the co-occurrence of the target vector and input melody is informed to the network, the binding force between them is not necessarily strong. On the other hand, in the case of the proposed method, the generator is forced to adapt to the “critics” from the discriminator. This
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Transitions of the MSE in the learning process of *D* with both training data and generated data (Equation (2)), *D* with generated data (the second term of Equation (2)), *D* with test data, and *G* with generated data (Equation (3)).</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>**Figure 2.** The transitions of the averages of $|r_i - t_i|$ for respective $i$ ($i = 1, 2, 3, 4$). These indicate to what extent respective rules are satisfied.</figcaption></figure>

*G* (Equation (3)) converged to 0.0, which implies that discriminations of both training data (D) and generated samples (D generated) were successful and *G* have acquired the capacity of satisfying the constraints (Equation (3)).

The loss of discrimination of the test data (D test data) continuously decreases, and the mean loss (MSE) is below 10%

Figure 2 shows the learning processes of respective rules (the transitions of the averages of $|r_i - t_i|$ for respective $i$ ($i = 1, 2, 3, 4$). Figure 3 shows the transitions of averages of $r_i$ for each $i$. This Figure indicates that all of the four elements of $R_\omega$ converge to the target values of $T$. We can observe that $r_1$ converges more accurately than others, which is understandable. The satisfaction of Rule 1 is easier because whether a note in a melody is inside the scale or not is independent of other notes.

Figure 4 shows some generated melodies whose loss was almost 0.0 (optimal). We can observe that the contours of these samples are similar. This is probably because the generator lost its diversity through the learning process and focused on specific area of optimal solutions.

<figure><img src="image_4.png" /><figcaption>**Figure 3.** The transitions of averages of $r_i$ for respective $i$ ($i = 1, 2, 3, 4$). These indicate that $R_\omega$ successfully converged to $T$ (the horizontal lines).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Some examples of generated melodies. These samples realize $R_\omega = T$ (= (1.0, 1.0, 0.66, 0.36)), which means the constraints are perfectly satisfied. We can observe that there is no note outside of a diatomic scale, that all melodic intervals are included in the recommended set of intervals, that there are many conjunct motions, and that there are not too many direction changes.</figcaption></figure>

## 4.4 Discussion

From the result of the experiment, we have confirmed that the proposed GAN method has a capacity for constraint satisfaction. More precisely, the discriminator *D* could learn each rule “separately” from each synthetic data set and could teach the generator *G* to obey multiple rules simultaneously.

Although the author has conducted a preliminary experiment with another method based on a conditional VAE of which the target vector *T* is used as the conditional input, it was not successful. An important difference between this method and the proposed method is whether there is a mechanism of discriminator or not. In the case of conditional VAE, although the co-occurrence of the target vector and input melody is informed to the network, the binding force between them is not necessarily strong. On the other hand, in the case of the proposed method, the generator is forced to adapt to the “critics” from the discriminator. This
mechanism of GAN's discriminator seems to contribute to
the success of constraint satisfaction.

This capacity of constraint satisfaction suggested in this
research may add another possibility of controlling the con-
tents of generative AI by introducing explicit rules and de-
grees of satisfying constraints. It may also contribute to
improving the trustfulness of generative AI, which does not
yet have sufficient ways of regulating unwanted results.

As an extension of this research, there are remaining re-
search issues. Although only synthetic data sets were used
in this study, there is a possibility of combining real data in
order to add the functionality of imitating real data as in the
original GAN. In addition, the proposed method has a limi-
tation that the target vector is fixed. If we change the target
T, we must train the neural network from the beginning.
To make the GAN possible to have more flexible control-
lability of the constraints, one imaginable modification is
to add the target vector T as a conditional input.

5. CONCLUSION

This paper proposed a new generalized GAN variant de-
signed to solve constraint satisfaction problems. The ex-
perimental result of the melodic generation task with some
musical constraints shows that the proposed GAN can learn
multiple rules separately from different data sets and gen-
erate instances that satisfy all the rules simultaneously.

There are some remaining research issues. An experi-
ment of combining imitation with real data and constraint
satisfaction with synthetic data should be done. This hy-
brid approach may lead to practical applications that need
to respect some explicit crucial constraints when generat-
ing instances similar to those in real data. In addition, it
is desirable to improve the proposed method so that it can
change the target constraint values more flexibly.

Acknowledgments

This work was supported by JST, ACT-X Grant Number
JPMJAX21AD, Japan.

6. REFERENCES

[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative Adversarial Nets," in *Advances in Neural Information Processing Systems*, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2014/ file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

[2] J. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks," *CoRR*, vol. abs/1703.10593, 2017. [Online]. Available: http://arxiv.org/abs/1703. 10593

[3] M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” *CoRR*, vol. abs/1411.1784, 2014. [Online]. Available: http://arxiv.org/abs/1411.1784

[4] T. Karras, S. Laine, and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” *CoRR*, vol. abs/1812.04948, 2018. [Online]. Available: http://arxiv.org/abs/1812.04948

[5] L. Yang, S. Chou, and Y. Yang, “MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation using 1D and 2D Conditions,” *CoRR*, vol. abs/1703.10847, 2017. [Online]. Available: http://arxiv.org/abs/1703.10847

[6] J. J. Fux, *The study of counterpoint from Johann Joseph Fux's Gradus ad parnassum*. WW Norton & Company, 1965.

[7] Y. Hiroshi, *Traité de contrepoint d'après les méthodes du conservatoire de Paris*. Ongakunotomosha, 2012.

[8] G. Noël and B. Marcel, *Traité de contrepoint*. Durand, 1964.

[9] W. Shottstaedt, “Automatic counterpoint,” in *Current directions in computer music research*, 1989.

[10] A. Torsten, C. Anagnostopoulou, and M. Alcorn, “Strasheela: design and usage of a music composition environment based on the Oz programming model,” in *Proceedings of International Conference on Multiparadigm Programming in Mozart/Oz*, Charleroi, 2005, pp. 277–291.

[11] M. Farbood and B. Schöner, “Analysis and synthesis of Palestrina-style counterpoint using Markov chains,” in *Proceedings of International Computer Music Conference*, 2001.

[12] T. Tanaka and et al., “Automatic Music Composition Based on Counterpoint and Imitation Using Stochastic Models,” in *Proceedings of 7th Sound and Music Computing Conference*, Barcelona, 2010, pp. 213–218.

[13] D. Herremans and K. Sörensen, “Composing first species counterpoint with a variable neighbourhood search algorithm,” in *Journal of Mathematics and the Arts* 6.4, 2012, pp. 169–189.

[14] T. Tanaka, “Formulating First Species Counterpoint With Integer Programming,” in *Proceedings of 19th Sound and Music Computing Conference*, Saint-Etienne, 2022, pp. 601–607.
mechanism of GAN's discriminator seems to contribute to
the success of constraint satisfaction.

This capacity of constraint satisfaction suggested in this
research may add another possibility of controlling the con-
tents of generative AI by introducing explicit rules and de-
grees of satisfying constraints. It may also contribute to
improving the trustfulness of generative AI, which does not
yet have sufficient ways of regulating unwanted results.

As an extension of this research, there are remaining re-
search issues. Although only synthetic data sets were used
in this study, there is a possibility of combining real data in
order to add the functionality of imitating real data as in the
original GAN. In addition, the proposed method has a limi-
tation that the target vector is fixed. If we change the target
T, we must train the neural network from the beginning.
To make the GAN possible to have more flexible control-
lability of the constraints, one imaginable modification is
to add the target vector T as a conditional input.

5. CONCLUSION

This paper proposed a new generalized GAN variant de-
signed to solve constraint satisfaction problems. The ex-
perimental result of the melodic generation task with some
musical constraints shows that the proposed GAN can learn
multiple rules separately from different data sets and gen-
erate instances that satisfy all the rules simultaneously.

There are some remaining research issues. An experi-
ment of combining imitation with real data and constraint
satisfaction with synthetic data should be done. This hy-
brid approach may lead to practical applications that need
to respect some explicit crucial constraints when generat-
ing instances similar to those in real data. In addition, it
is desirable to improve the proposed method so that it can
change the target constraint values more flexibly.

Acknowledgments

This work was supported by JST, ACT-X Grant Number
JPMJAX21AD, Japan.

6. REFERENCES

[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in *Advances in Neural Information Processing Systems*, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2014/ file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

[2] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” *CoRR*, vol. abs/1703.10593, 2017. [Online]. Available: http://arxiv.org/abs/1703. 10593

[3] M. Mirza and S. Osindero, “Conditional Generative Adversarial Nets,” CoRR, vol. abs/1411.1784, 2014. [Online]. Available: http://arxiv.org/abs/1411.1784

[4] T. Karras, S. Laine, and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CoRR, vol. abs/1812.04948, 2018. [Online]. Available: http://arxiv.org/abs/1812.04948

[5] L. Yang, S. Chou, and Y. Yang, “MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation using 1D and 2D Conditions,” CoRR, vol. abs/1703.10847, 2017. [Online]. Available: http://arxiv.org/abs/1703.10847

[6] J. J. Fux, *The study of counterpoint from Johann Joseph Fux's Gradus ad parnassum*. WW Norton & Company, 1965.

[7] Y. Hiroshi, *Traité de contrepoint d'après les méthodes du conservatoire de Paris*. Ongakunotomosha, 2012.

[8] G. Noël and B. Marcel, *Traité de contrepoint*. Durand, 1964.

[9] W. Shottstaedt, “Automatic counterpoint,” in *Current directions in computer music research*, 1989.

[10] A. Torsten, C. Anagnostopoulou, and M. Alcorn, “Strasheela: design and usage of a music composition environment based on the Oz programming model,” in *Proceedings of International Conference on Multiparadigm Programming in Mozart/Oz*, Charleroi, 2005, pp. 277–291.

[11] M. Farbood and B. Schöner, “Analysis and synthesis of Palestrina-style counterpoint using Markov chains,” in *Proceedings of International Computer Music Conference*, 2001.

[12] T. Tanaka and et al., “Automatic Music Composition Based on Counterpoint and Imitation Using Stochastic Models,” in *Proceedings of 7th Sound and Music Computing Conference*, Barcelona, 2010, pp. 213–218.

[13] D. Herremans and K. Sörensen, “Composing first species counterpoint with a variable neighbourhood search algorithm,” in *Journal of Mathematics and the Arts 6.4*, 2012, pp. 169–189.

[14] T. Tanaka, “Formulating First Species Counterpoint With Integer Programming,” in *Proceedings of 19th Sound and Music Computing Conference*, Saint-Etienne, 2022, pp. 601–607.