CocoChorales

# The CocoChorales Dataset

**CocoChorales** is a dataset consisting of over 1400 hours of audio mixtures containing four-part chorales performed by 13 instruments, all synthesized with realistic-sounding generative models. CocoChorales contains mixes, sources, and MIDI data, as well as annotations for note expression (e.g., per-note volume and vibrato) and synthesis parameters (e.g., multi-f0).

## Dataset

We created CocoChorales using two generative models produced by Magenta: [Coconet](https://magenta.tensorflow.org/coconet) and [MIDI-DDSP](https://magenta.tensorflow.org/midi-ddsp). The dataset was created in two stages. First, we used a trained Coconet model to generate a large set of four-part chorales in the style of J.S. Bach. The output of this first stage is a set of note sequences, stored as MIDI, to which we assign a tempo and add random timing variations to each note (for added realism).

In the second stage, we use MIDI-DDSP to synthesize these MIDI files into audio, resulting in audio clips that sound like the chorales were performed by live musicians. This MIDI-DDSP model was trained on [URMP](https://labsites.rochester.edu/air/projects/URMP.html). We define a set of ensembles that consist of the following instruments, in Soprano, Alto, Tenor, Bass (SATB) order:

<ul>
  <li><strong>String Ensemble</strong>: Violin 1, Violin 2, Viola, Cello.</li>
  <li><strong>Brass Ensemble</strong>: Trumpet, French Horn, Trombone, Tuba.</li>
  <li><strong>Woodwind Ensemble</strong>: Flute, Oboe, Clarinet, Bassoon.</li>
  <li><strong>Random Ensemble</strong>: Each SATB part is randomly assigned an instrument according to the following:
    <ul class="nested">
      <li><em>Soprano</em>: Violin, Flute, Trumpet, Clarinet, Oboe.</li>
      <li><em>Alto</em>: Violin, Viola, Flute, Clarinet, Oboe, Saxophone, Trumpet, French Horn.</li>
      <li><em>Tenor</em>: Viola, Cello, Clarinet, Saxophone, Trombone, French Horn.</li>
      <li><em>Bass</em>: Cello, Double Bass, Bassoon, Tuba.</li>
    </ul>
  </li>
</ul>


Each instrument in the ensemble is synthesized separately, with annotations for the high-level expressions used for each note (e.g., vibrato, note volume, note brightness, etc; all expressions shown [here](https://midi-ddsp.github.io/#note_expression_control), and more details in Sections 3.2 and B.3 of the [MIDI-DDSP paper](https://openreview.net/pdf?id=UseMOjWENv)) as well as detailed low-level annotations for synthesis parameters (e.g., f<sub>0</sub>’s, amplitudes of each harmonic, etc). Because the MIDI-DDSP model skews sharp, we randomly applied pitch augmentation to the f<sub>0</sub>’s (see Figure 2, [here](https://arxiv.org/pdf/2209.14458.pdf)) to . All four audio clips for each instrument in the ensemble are then mixed together to produce an example in the dataset.

Because all of the data in CocoChorales originate from generative models, all of the annotations perfectly correspond to the audio data. All in all, the dataset contains 240,000 examples, 60,000 mixes from each one of the four ensemble types above. Each ensemble has its own train/validation/test split All of the audio is 16 kHz, 16-bit PCM data. Each example contains:


<ul>
  <li>A mixture</li>
  <li>Source audio for all four instruments
    <ul class="nested">
      <li>Gain applied to each source</li>
    </ul>
  </li>
  <li>MIDI with tempo and precise timing</li>
  <li>The name of the ensemble with instrument names</li>
  <li>Note expression annotations for every note:
    <ul class="nested">
      <li>Volume, Volume Fluctuation, Volume Peak Position, Vibrato, Brightness, and Attack Noise used by MIDI-DDSP to synthesize every note (see Sections 3.2 and B.3 of the MIDI-DDSP paper for more details)</li>
    </ul>
  </li>
  <li>Synthesis parameters for every source (250 Hz):
    <ul class="nested">
      <li>Fundamental frequency (f<sub>0</sub>), amplitude, amplitude of all harmonics, filtered noise parameters</li>
      <li>Amount of pitch augmentation applied</li>
    </ul>
  </li>
</ul>


## Further Details

A detailed view of the contents of the CocoChorales dataset is provided [at this link](https://github.com/lukewys/chamber-ensemble-generator/blob/master/data_format.md).

# Download

For download instructions, please see [this github page](https://github.com/lukewys/chamber-ensemble-generator#dataset-download). The compressed version of the full dataset is 2.9 Tb, and the uncompressed version is larger than 4 Tb. There is a "tiny" version for download as well.

MD5 Hashes for all zipped files in the download are provided [here](https://storage.googleapis.com/magentadata/datasets/cocochorales/cocochorales_full_v1_zipped/cocochorales_md5s.txt).


# License

The CocoChorales dataset was made by [Yusong Wu](https://lukewys.github.io/) and is available under the [Creative Commons Attribution 4.0 International (CC-BY 4.0)](https://creativecommons.org/licenses/by/4.0/).


# How to Cite

If you use CocoChorales in your work, we ask that you cite the following [paper](https://arxiv.org/abs/2209.14458) where it was introduced:

```
Yusong Wu, Josh Gardner, Ethan Manilow, Ian Simon, Curtis Hawthorne, and Jesse Engel.
“The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling.”
arXiv preprint, arXiv:2209.14458, 2022.
```

You can also use the following bibtex entry:

```
@article{wu2022chamber,
  title = {The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling},
  author = {Wu, Yusong and Gardner, Josh and Manilow, Ethan and Simon, Ian and Hawthorne, Curtis and Engel, Jesse},
  journal={arXiv preprint arXiv:2209.14458},
  year = {2022},
}
```

Introducing paper: The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling