Add CMU MoCap Dataset
State: closed
Labels: description: Contribution to a dataset script
name: dataset contribution

Resolves #3457 

Dataset Request : Add CMU Graphics Lab Motion Capture dataset [#3457](https://github.com/huggingface/datasets/issues/3457)
This PR adds the CMU MoCap Dataset.

The authors didn't respond even after multiple follow ups, so I ended up crawling the website to get categories, subcategories and description information. Some of the subjects do not have category/subcategory/description as well. I am using a subject to categories, subcategories and description map (metadata file).

Currently the loading of the dataset works for "asf/amc" and "avi" formats since they have a single download link. But "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Any suggestions/inputs on this would be helpful. Thank you.




Comments:
_The documentation is not available anymore as the PR was closed or merged._

- Updated the readme.
- Added dummy_data.zip and ran the all the tests.

The dataset works for "asf/amc" and "avi" formats which have a single download link for the complete dataset. But "c3d" and "mpg" have multiple download links, can we combine and host these links on the Hub since the dataset is free to use ?

"c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Can we combine and host these links on the Hub since the dataset is free to use ?

> "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?

We store downloaded data under `~/.cache/huggingface/datasets/downloads` (by default), so these downloads are "hidden" and won't clutter one's filesystem in an "obvious way".

> We store downloaded data under ~/.cache/huggingface/datasets/downloads (by default), so these downloads are "hidden" and won't clutter one's filesystem in an "obvious way".

Yes, the filesystem won't be clustered, but the problem is processing the dataset becomes cumbersome. For eg, for the c3d format has 5 part-downloads, so the folders will be as follows : 
```
['~/.cache/huggingface/datasets/downloads/extracted/0e6bf028f490bf18c23ce572d1437c4ef32a74f630e33c26a806250d35cfcdd1', '~/.cache/huggingface/datasets/downloads/extracted/1b44fc5c7a6e031c904545422d449fd964f8ee795b9d1dcb0b6a76d03b50ebe6', '~/.cache/huggingface/datasets/downloads/extracted/137595188e96187c24ce1aa5c78200c7f78816fbd9d6c62354c01b3e6ec550c7', '~/.cache/huggingface/datasets/downloads/extracted/6c0c893e435f36fd79aa0f199f58fe16f01985f039644a7cb094a8c43a15ffd4', '~/.cache/huggingface/datasets/downloads/extracted/45e4703354cbc975e6add66f1b17b716c882b56f44575b033c5926aa5fcfb17f']
```
Each of these folders have a given set of subjects, so we'll be need to write extra code to fetch data from each of these folders, and the mpg format has 12 part-downloads which will lead to 12 folders having certain set of subjects, so it is cumbersome to process them.

I have added all the changes that were suggested. We just need to handle the multi-part download for c3d and mpg formats. Easiest way would be to have just one zip for these formats.

But we can handle this with a simple mapping that stores the id ranges (for each config), no? And an actual file path is not important during processing.

I have added code to handle c3d, mpg formats as well.  The data for the mpg format seems incomplete as it contains only 53 rows. I have added a note regarding this in the Data Splits section.

The real data test works fine and dummy_data test work fine. There were few missing files which was causing issues, I have fixed it now.


- Reduced the dummy_data size.
- Added sample dataset preprocessing code, it is not complete though.
- Added all changes suggested.

Let me know if anything else is required. Thank you. :)

Thanks for your contribution, @dnaveenr.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.