Package: rouskinhf
Download your RNA data from HuggingFace with rouskinhf!A repo to manipulate the data for our RNA structure prediction model. This repo allows you to:pull datasets from the Rouskinlab's HuggingFacecreate datasets from local files and push them to HuggingFace, from the formats:.fasta.ct.json(DREEM output format).json(Rouskinlab's huggingface format)Important notesSequences with bases different thanA,C,G,T,U,N,a,c,g,t,u,nare not supported. The data will be filtered out.DependenciesRNAstructure(also available onRouskinlab GitHub).Push a new release to PypiEdit version tovx.y.zinpyproject.toml. Then run in a terminalgit add . && git commit -m 'vx.y.z' && git push.Create and push a git tagvx.y.zby running in a terminalgit tag 'vx.y.z' && git push --tag.Create a release for the tagvx.y.zon Github Release.Make sure that the Github ActionPublish distributions 📦 to PyPIpassed on Github Actions.InstallationGet a HuggingFace tokenGo toHuggingFaceand create an account. Then go to your profile and copy your token (huggingface.co/settings/tokens).Create an environment fileOpen a terminal and type:nanoenvCopy paste the following content, and change the values to your own:exportHUGGINGFACE_TOKEN="your token here"# you must change this to your HuggingFace tokenexportDATA_FOLDER="data/datafolders"# where the datafolder are stored by default, change it if you want to store it somewhere elseexportDATA_FOLDER_TESTING="data/input_files_for_testing"# Don't touch thisexportRNASTRUCTURE_PATH="/Users/ymdt/src/RNAstructure/exe"# Change this to the path of your RNAstructure executableexportRNASTRUCTURE_TEMP_FOLDER="temp"# You can change this to the path of your RNAstructure temp folderThen save the file and exit nano.Source the environmentsourceenvInstall the package with pippipinstallrouskinhfTutorialsAuthentify your machine to HuggingFaceSee thetutorial.Download a datafolder from HuggingFaceSee thetutorial.Create a datafolder from local files and push it to HuggingFaceSee thetutorial.AboutSourcing the environment and keeping your environment variable secretThe variables defined in theenvfile are required byrouskinhf. Make that before you userouskinhf, you run in a terminal:sourceenvor, in a Jupyter notebook:!pipinstallpython-dotenv%load_extdotenv%dotenvenvor, in a python script or Jupyter notebook:fromrouskinhfimportsetup_envsetup_env(HUGGINGFACE_TOKEN="your token here",DATA_FOLDER="data/datafolders",...)The point of using environment variables is to ensure the privacy of your huggingface token. Make sure to add yourenvfile to your.gitignore, so your HuggingFace token doesn't get pushed to any public repository.Import data withimport_datasetThis repo provides a functionimport_dataset, which allows your to pull a dataset from HuggingFace and store it locally. If the data is already stored locally, it will be loaded from the local folder. The type of data available is the DMS signal and the structure, under the shape of paired bases tuples. The function has the following signature:defimport_dataset(name:str,data:str,force_download:bool=False)->np.ndarray:"""Finds the dataset with the given name for the given type of data.Parameters----------name : strName of the dataset to find.data : strName of the type of data to find the dataset for (structure or DMS).force_download : boolWhether to force download the dataset from HuggingFace Hub. Defaults to False.Returns-------ndarrayThe dataset with the given name for the given type of data.Example------->>> import_dataset(name='for_testing', data='structure').keys()dict_keys(['references', 'sequences', 'structure'])>>> import_dataset(name='for_testing', data='DMS').keys()dict_keys(['references', 'sequences', 'DMS'])>>> import_dataset(name='for_testing', data='structure', force_download=True).keys()dict_keys(['references', 'sequences', 'structure'])>>> import_dataset(name='for_testing', data='DMS', force_download=True).keys()dict_keys(['references', 'sequences', 'DMS'])FYI, the datafolder objectThe datafolder object is a wrapper around your local folder and HuggingFace API, to keep a consistent datastructure across your datasets. It contains multiple methods to create datasets from various input formats, store the data and metadata in a systematic way, and push / pull from HuggingFace.On HuggingFace, the datafolder stores the data under the following structure:HUGGINGFACEDATAFOLDER
-[datafoldername]-source-whicheverfile(s)youusedtocreatethedataset(fasta,setofCTs,etc.).-data.json# the data under a human readable format.-info.json# the metadata of the dataset. This file indicates how we got the DMS signal and the structures (directly from the source or from a prediction).-README.md# the metadata of the dataset in a human readable format.Locally, we have the same structure with the addition of .npy files which contain the data in a machine readable format. Each .npy file contains a numpy array of the data, and the name of the file is the name of the corresponding key in the data.json file. The source file won’t be downloaded by default. Hence, the local structure is:LOCALDATAFOLDER
-[datafoldername]...-README.md# the metadata of the dataset in a human readable format-references.npy-sequences.npy-base_pairs.npy-dms.npy