Repository: mir-dataset-loaders/mirdata

Issue:
dali._load_metadata() points to the wrong path
Follow-up from a discussion with @gabolsgabs
Here are our instructions on how to download DALI:

```

        To download this dataset, visit:
        https://zenodo.org/record/2577915 and request access.

        Once downloaded, unzip the file DALI_v1.0.zip
        and place the result in:
        /Users/vl238/mir_datasets/DALI/DALI/annotatios

        Use the function dali_code.get_audio you can find at:
        https://github.com/gabolsgabs/DALI for getting the audio and place them at:
        /Users/vl238/mir_datasets/DALI/DALI/audio
```

First, the path is wrong. It should be `/Users/vl238/mir_datasets/DALI/annotatios`, not `/Users/vl238/mir_datasets/DALI/annotations` (note the two `DALI` folders, and the typo in `annotatios`)

The problem is that this Zenodo record does not have metadata. So this will work: (because it's a property)

```
dali.load()['001940b614eb43f4a0c826d49a67d66d'].words
```

This won't work (because it's an attribute)
```
dali.load()['001940b614eb43f4a0c826d49a67d66d'].album
```

Luckily, @gabolsgabs has agreed to host the metadata JSON file on GitHub:
https://github.com/gabolsgabs/DALI/blob/master/code/DALI/files/dali_v1_metadata.json

He release DALI v1.1 with this JSON file. So from the scope of mirdata, we should be able to require DALI v1.1 and load that file automatically in the `mirdata.dali._load_metadata`

@rabitt i suppose this is urgent?

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/mir-dataset-loaders/mirdata/pull/230?src=pr&el=h1) Report
> Merging [#230](https://codecov.io/gh/mir-dataset-loaders/mirdata/pull/230?src=pr&el=desc) into [master](https://codecov.io/gh/mir-dataset-loaders/mirdata/commit/bfbe6ccafaf87d85dbd213f1da2bcb2be3e19e9f&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `50.00%`.


```diff
@@            Coverage Diff             @@
##           master     #230      +/-   ##
==========================================
- Coverage   95.91%   95.86%   -0.05%     
==========================================
  Files          20       20              
  Lines        2080     2082       +2     
==========================================
+ Hits         1995     1996       +1     
- Misses         85       86       +1     
```

lostanlen: This is ready for review. Note that i ended up using a `raw.githubusercontent.com` address, not a `github.com` address, because the latter kept giving me inconsistent checksums
rabitt: Before reviewing - you're implementing `dali.Track.to_jams(duration)`, but we didn't yet decide on #226 . Let's wait for that decision first?
lostanlen: @rabitt i implemented `dali.Track.to_jams()` by looking up the duration of the audio. This is ready for CR

Patch:
diff --git a/docs/source/datasets.rst b/docs/source/datasets.rst
index ad7daf4..f58837a 100644
--- a/docs/source/datasets.rst
+++ b/docs/source/datasets.rst
@@ -33,8 +33,8 @@ click the link provided in the "Module" column.
 | dali_            | DALI                | - audio: 📺         | - :ref:`lyrics`           | 5358   |
 |                  |                     | - annotations: ✅   | - Vocal :ref:`notes`      |        |
 +------------------+---------------------+---------------------+---------------------------+--------+
-| groove_midi_     | Groove MIDI         | - audio: ✅         | - :ref:`beats`            | 1150   |
-|                  | Dataset             | - midi: ✅          | - :ref:`tempo`            |        |
+| groove_midi_     | | Groove MIDI       | - audio: ✅         | - :ref:`beats`            | 1150   |
+|                  | | Dataset           | - midi: ✅          | - :ref:`tempo`            |        |
 |                  |                     |                     | - :ref:`drums`            |        |
 +------------------+---------------------+---------------------+---------------------------+--------+
 | gtzan_genre_     | Gtzan-Genre         | - audio: ✅         | - :ref:`genre`            | 1000   |
@@ -45,6 +45,7 @@ click the link provided in the "Module" column.
 |                  |                     |                     | - :ref:`key`              |        |
 |                  |                     |                     | - :ref:`notes`            |        |
 |                  |                     |                     | - :ref:`f0`               |        |
+|                  |                     |                     | - :ref:`tempo`            |        |
 +------------------+---------------------+---------------------+---------------------------+--------+
 | ikala_           | iKala               | - audio: ❌         | - Vocal :ref:`F0`         | 252    |
 |                  |                     | - annotations: ❌   | - :ref:`lyrics`           |        |
@@ -71,6 +72,7 @@ click the link provided in the "Module" column.
 |                  |                     | - annotations: ✅   | - :ref:`sections`         |        |
 |                  |                     |                     | - :ref:`vocal-activity`   |        |
 |                  |                     |                     | - :ref:`chords`           |        |
+|                  |                     |                     | - :ref:`tempo`            |        |
 +------------------+---------------------+---------------------+---------------------------+--------+
 | salami_          | Salami              | - audio: ❌         | - :ref:`sections`         | 1359   |
 |                  |                     | - annotations: ✅   |                           |        |
@@ -94,7 +96,7 @@ the dataset specific documentation to ensure the data is as you expect.
 
 Beats
 ^^^^^
-Musical beats, typically encoded as sequence of time stamps and corresponding beat positions.
+Musical beats, typically encoded as sequence of timestamps and corresponding beat positions.
 This implicitly includes *downbeat* information (the beginning of a musical measure).
 
 .. _chords:
@@ -118,7 +120,7 @@ one another, as multiple drums can be played at the same time.
 F0
 ^^
 Musical pitch contours, typically encoded as time series indidcating the musical pitch over time.
-The time series typically have evenly spaced time stamps, each with a correspoinding pitch value
+The time series typically have evenly spaced timestamps, each with a correspoinding pitch value
 which may be encoded in a number of formats/granularities, including midi note numbers and Hertz.
 
 .. _genre:
@@ -166,6 +168,14 @@ Musical note events, typically encoded as sequences of start time, end time, lab
 indicates a musical pitch, which may be in a number of formats/granularities, including midi note numbers,
 Hertz, or pitch class.
 
+.. _sections:
+
+Sections
+^^^^^^^^
+Musical sections, which may be "flat" or "hierarchical", typically encoded by a sequence of
+timestamps indicating musical section boundary times. Section annotations sometimes also
+include labels for sections, which may indicate repetitions and/or the section type (e.g. Chorus, Verse).
+
 .. _technique:
 
 Technique
@@ -173,13 +183,14 @@ Technique
 The playing technique used by a particular instrument, for example "Pizzicato". This label may be global
 for a given recording or encoded as a sequence of labeled events.
 
-.. _sections:
+.. _tempo:
 
-Sections
-^^^^^^^^
-Musical sections, which may be "flat" or "heirarchical", typically encoded by a sequence of
-time stamps indicating musical section boundary times. Section annotations sometimes also
-include labels for sections, which may indicate repetitions and/or the section type (e.g. Chorus, Verse).
+Tempo
+^^^^^
+The tempo of a song, typical in units of beats-per-minute (bpm). This is often indicated globally per track,
+but in practice tracks may have tempos that change, and some datasets encode tempo as time-varying quantity.
+Additionally, there may be multiple reasonable tempos at any given time (for example, often 2x or 0.5x a
+tempo value will also be "correct"). For this reason, some datasets provide two or more different tempo values.
 
 .. _vocal-activity:
 
@@ -192,6 +203,7 @@ of annotation is implicitly available when Vocal F0_ or Vocal Notes_ annotations
 .. _article: https://link.springer.com/article/10.1007/s10844-013-0250-y
 .. _beatles: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.beatles
 .. _dali: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.dali
+.. _groove_midi: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.groove_midi
 .. _gtzan_genre: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.gtzan_genre
 .. _guitarset: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.guitarset
 .. _ikala: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.ikala
@@ -204,7 +216,7 @@ of annotation is implicitly available when Vocal F0_ or Vocal Notes_ annotations
 .. _rwc_pop: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.rwc_pop
 .. _salami: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.salami
 .. _tinysol: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.tinysol
-.. _groove_midi: https://mirdata.readthedocs.io/en/latest/source/mirdata.html#module-mirdata.groove_midi
+
 
 
 
diff --git a/mirdata/dali.py b/mirdata/dali.py
index e7597bf..58b2499 100644
--- a/mirdata/dali.py
+++ b/mirdata/dali.py
@@ -41,6 +41,13 @@ import mirdata.utils as utils
 
 DATASET_DIR = 'DALI'
 
+METADATA_REMOTE = download_utils.RemoteFileMetadata(
+    filename='dali_metadata.json',
+    url='https://raw.githubusercontent.com/gabolsgabs/DALI/master/code/DALI/files/dali_v1_metadata.json',
+    checksum='40af5059e7aa97f81b2654758094d24b',
+    destination_dir='.',
+)
+
 
 def _load_metadata(data_home):
     metadata_path = os.path.join(data_home, os.path.join('dali_metadata.json'))
@@ -166,8 +173,20 @@ class Track(track.Track):
         return load_audio(self.audio_path)
 
     def to_jams(self):
-        """(Not Implemented) Jams: the track's data in jams format"""
-        raise NotImplementedError
+        """Jams: the track's data in jams format"""
+        # Load metadata
+        metadata = {k: v for k, v in self._track_metadata.items() if v is not None}
+        y, sr = self.audio
+        metadata['duration'] = librosa.get_duration(y=y, sr=sr)
+        return jams_utils.jams_converter(
+            lyrics_data=[
+                (self.words, 'word-aligned lyrics'),
+                (self.lines, 'line-aligned lyrics'),
+                (self.paragraphs, 'paragraph-aligned lyrics'),
+            ],
+            note_data=[(self.notes, 'annotated vocal notes')],
+            metadata=metadata,
+        )
 
 
 def load_audio(audio_path):
@@ -184,7 +203,7 @@ def load_audio(audio_path):
     return librosa.load(audio_path, sr=None, mono=True)
 
 
-def download(data_home=None):
+def download(data_home=None, force_overwrite=False):
     """DALI is not available for downloading directly.
     This function prints a helper message to download DALI
     through zenodo.org.
@@ -197,8 +216,7 @@ def download(data_home=None):
     if data_home is None:
         data_home = utils.get_default_dataset_path(DATASET_DIR)
 
-    print(
-        """
+    info_message = """
         To download this dataset, visit:
         https://zenodo.org/record/2577915 and request access.
 
@@ -209,11 +227,15 @@ def download(data_home=None):
         Use the function dali_code.get_audio you can find at:
         https://github.com/gabolsgabs/DALI for getting the audio and place them at:
         {audio_path}
-
     """.format(
-            save_path=os.path.join(data_home, DATASET_DIR, 'annotatios'),
-            audio_path=os.path.join(data_home, DATASET_DIR, 'audio'),
-        )
+        save_path=os.path.join(data_home, 'annotations'),
+        audio_path=os.path.join(data_home, 'audio'),
+    )
+    download_utils.downloader(
+        data_home,
+        file_downloads=[METADATA_REMOTE],
+        info_message=info_message,
+        force_overwrite=force_overwrite,
     )
 
 
diff --git a/mirdata/groove_midi.py b/mirdata/groove_midi.py
index cdc0fe9..a08022b 100644
--- a/mirdata/groove_midi.py
+++ b/mirdata/groove_midi.py
@@ -204,7 +204,7 @@ def _load_metadata(data_home):
                 'session': str(session),
                 'track_id': str(track_id),
                 'style': str(style),
-                'bpm': int(bpm),
+                'tempo': int(bpm),
                 'beat_type': str(beat_type),
                 'time_signature': str(time_signature),
                 'midi_filename': str(midi_filename),
@@ -234,7 +234,7 @@ class Track(track.Track):
         session (str): Type of session  (ex. 'session1', 'eval_session')
         track_id (str): track id of the track (ex. 'drummer1/eval_session/1')
         style (str): Style (genre, groove type) of the track (ex. 'funk/groove1')
-        bpm (int): Track bpm (ex. 138)
+        tempo (int): Track tempo in beats per minute (ex. 138)
         beat_type (str): Whether the track is a beat or a fill (ex. 'beat')
         time_signature (str): Time signature of the track (ex. '4-4', '6-8')
         midi_path (str): Path to the midi file
@@ -266,7 +266,7 @@ class Track(track.Track):
                 "drummer": None,
                 "session": None,
                 "style": None,
-                "bpm": None,
+                "tempo": None,
                 "beat_type": None,
                 "time_signature": None,
                 "midi_filename": None,
@@ -278,7 +278,7 @@ class Track(track.Track):
         self.drummer = self._track_metadata["drummer"]
         self.session = self._track_metadata["session"]
         self.style = self._track_metadata["style"]
-        self.bpm = self._track_metadata["bpm"]
+        self.tempo = self._track_metadata["tempo"]
         self.beat_type = self._track_metadata["beat_type"]
         self.time_signature = self._track_metadata["time_signature"]
         self.duration = self._track_metadata["duration"]
@@ -331,7 +331,7 @@ class Track(track.Track):
         # Initialize top-level JAMS container
         return jams_utils.jams_converter(
             beat_data=[(self.beats, 'midi beats')],
-            tempo_data=[(self.bpm, 'midi tempo')],
+            tempo_data=[(self.tempo, 'midi tempo')],
             event_data=[(self.drum_events, 'annotated drum patterns')],
             metadata=self._track_metadata,
         )
diff --git a/setup.py b/setup.py
index 3661a67..9fa0c2d 100644
--- a/setup.py
+++ b/setup.py
@@ -48,16 +48,15 @@ if __name__ == '__main__':
                 'testcontainers>=2.3',
                 'future==0.17.1',
                 'coveralls>=1.7.0',
-                'dali-dataset',
             ],
             'docs': [
-                'DALI-dataset>=1.0.0',
+                'dali-dataset==1.1',
                 'numpydoc',
                 'recommonmark',
                 'sphinx',
                 'sphinxcontrib-napoleon',
                 'sphinx_rtd_theme',
             ],
-            'dali': ['DALI-dataset>=1.0.0'],
+            'dali': ['dali-dataset==1.1'],
         },
     )


Test patch:
diff --git a/tests/test_groove_midi.py b/tests/test_groove_midi.py
index a64a06a..f9f70e7 100644
--- a/tests/test_groove_midi.py
+++ b/tests/test_groove_midi.py
@@ -15,7 +15,7 @@ def test_track():
         'session': 'drummer1/eval_session',
         'track_id': 'drummer1/eval_session/1',
         'style': 'funk/groove1',
-        'bpm': 138,
+        'tempo': 138,
         'beat_type': 'beat',
         'time_signature': '4-4',
         'midi_filename': 'drummer1/eval_session/1_funk-groove1_138_beat_4-4.mid',
@@ -68,7 +68,7 @@ def test_load_metadata():
         'session': 'drummer1/eval_session',
         'track_id': 'drummer1/eval_session/1',
         'style': 'funk/groove1',
-        'bpm': 138,
+        'tempo': 138,
         'beat_type': 'beat',
         'time_signature': '4-4',
         'midi_filename': 'drummer1/eval_session/1_funk-groove1_138_beat_4-4.mid',
diff --git a/tests/test_loaders.py b/tests/test_loaders.py
index 464222c..ab9d682 100644
--- a/tests/test_loaders.py
+++ b/tests/test_loaders.py
@@ -14,6 +14,7 @@ from tests.test_utils import DEFAULT_DATA_HOME
 DATASETS = [importlib.import_module("mirdata.{}".format(d)) for d in mirdata.__all__]
 CUSTOM_TEST_TRACKS = {
     'beatles': '0111',
+    'dali': '4b196e6c99574dd49ad00d56e132712b',
     'guitarset': '03_BN3-119-G_solo',
     'medley_solos_db': 'd07b1fc0-567d-52c2-fef4-239f31c9d40e',
     'medleydb_melody': 'MusicDelta_Beethoven',
@@ -67,7 +68,6 @@ def test_load_and_trackids():
 
 
 def test_track():
-    jams_temporary_exceptions = ['dali']
     data_home_dir = 'tests/resources/mir_datasets'
 
     for dataset in DATASETS:
@@ -92,10 +92,9 @@ def test_track():
 
         assert hasattr(track_test, 'to_jams')
 
-        if dataset_name not in jams_temporary_exceptions:
-            # Validate json schema
-            jam = track_test.to_jams()
-            assert jam.validate()
+        # Validate JSON schema
+        jam = track_test.to_jams()
+        assert jam.validate()
 
         # will fail if something goes wrong with __repr__
         print(track_test)