Package: tuncat
TUnCaTTemporal Unmixing of Calcium Traces (TUnCaT) is an automatic algorithm to decontaminate false transients from temporal traces generated from fluorescent calcium imaging videos. TUnCaT removed false transients caused by large-scale background fluctuation using background subtraction; TUnCaT removed false transients caused by neighboring neurons, axons, and dendrites using nonnegative matrix factorization (NMF).Copyright (C) 2021 Duke University NeuroToolboxThis repo contains the code of TUnCaT. If you want to reproduce the results in our paper, please visit thepaper reproduction repoorfigshareto find the data, results, and code to analyze the results.TUnCaTInstallation on Windows or LinuxDemoInput, Output, and Intermediate FilesInput filesIntermediate and Output filesUse your own dataUse your own videosUse your own neuron masksSet the unmixing parametersSet alpha as a user-defined value or using cross-validationCitationLicensing and CopyrightSponsorsInstallation on Windows or LinuxFirst, install Python from theofficial websiteor any other distribution likeAnaconda.Launch Windows/Linux terminal or Anaconda prompt, and install TUnCaT using pip:python -m pip install tuncatPlease be aware that the speed of TUnCaT in the Anaconda installation may be significantly slower than the speed in the offical python installation in Windows.DemoWe provided a demo for all users to get familiar with TUnCaT. To run the demo, please download the code from our GitHub repository. We provided a one-photon imaging videoc28_163_244.h5as well as its manually labeled neuronsFinalMasks_c28_163_244.mat. The demo will calculate the raw traces and background traces of all neurons, calculate the unmixed traces using TUnCaT, and export them to the folderunmixed_traces. The input, output, and intermediate files will be explained inInput, Output, and Intermediate Files.To run the demo python script, launch system terminal or Anaconda prompt, and type the following scriptcd{TUnCaT_root_path}
python demo_TUnCaT.pyYou can also run the demodemo_TUnCaT.ipynbinteractively using Jupyter Notebook. This notebook contains the expected figure of the background-subtracted trace and the unmixed trace of the first neuron. The saved processing time is recorded in a laptop with an AMD Ryzen 5 3500U quad-core CPU.Input, Output, and Intermediate FilesBy default, all the input, output, and intermediate files are saved under theTUnCaTfolder.Input filesA .h5 file{name}.h5contains the input video. This is a 3D dataset with shape = (T, Lx, Ly), where T is the number of frames, and (Lx, Ly) is the lateral dimension of each frame. The demo video has (T, Lx, Ly) = (6000, 50, 50).A .mat fileFinalMasks_{name}.matcontains the nueron masks of the video. This is a 3D array with shape = (Ly, Lx, n) in MATLAB, where Ly and Lx should match the lateral dimensions of the video, and n is the number of neurons. The demo mask has (Ly, Lx, n) = (50, 50, 45).Important notice: The default dimension order for multi-dimensional array is reversed in MATLAB and python. When you save a dataset with shape = (L1, L2, L3) in MATLAB to an .h5 file or a .mat file with version 7.3 or newer (requiring h5py.File to load in python workspace), and then load it in python, the shape will become (L3, L2, L1). However, if you save the dataset as a .mat file with version 7 or earlier (requiring scipy.loadmat to load in python workspace), the dimensions will preserve and still be (L1, L2, L3). In this document, we will use the python default order to describe the datasets in python workspace or saved in .h5, and use the MATLAB default order to describe the datasets saved in .mat. Sometimes you need to transpose the dimensions to make them consistent. In python, you can transpose the dimensions usingMasks = Masks.transpose((2,1,0)). In MATLAB, you can transpose the dimensions usingMasks = permute(Masks,[3,2,1]).Intermediate and Output filesAfter running TUnCaT on the demo video, the intermediate and output files will be under a new folderunmixed_traces.Intermediate file:unmixed_traces/raw/{name}.matstores the raw neuron traces (traces) and the background traces (bgtraces). Each trace variable is 2D matrix with shape = (T, n), where T is the number of frames, and n is the number of neurons.Output file:unmixed_traces/alpha={}/{name}.matorunmixed_traces/list_alpha={}/{name}.matstores the unmixed traces of the neurons (traces_nmfdemix), as well as other quantities characterizing the NMF unmixing process of each neuron, including the mixing matrix (list_mixout), final alpha (list_alpha_final), reconstruction residual (list_MSE), and number of iterations (list_n_iter). See the function descriptions for the detailed meanings of these quantities.Use your own dataOf course, you can modify the demo scripts to process other videos. You need to set the folders of the videos and neuron masks, and change some parameters in the python scripts to correspond to your videos.Use your own videosSet the folder and file names of your video correctly. You need to change the variablesdir_videoandlist_Exp_ID. The variabledir_videois the folder containing the input videos. For example, if your videos are stored inC:/Users/{username}/Documents/GitHub/TUnCaT_paper_reproduction/TUnCaT/data, setdir_video = 'C:/Users/{username}/Documents/GitHub/TUnCaT_paper_reproduction/TUnCaT/data'. You can also use relative path, such asdir_video = './data'. The variablelist_Exp_IDis the list of the file names (without extension) of the input videos (e.g.,list_Exp_ID = ['c28_163_244']in the demo referes to the input file{TUnCaT_root_path}/data/c28_163_244.h5).Currently, we only support .h5 files as the input video, so you need to convert the format of your data to .h5. You can save the video in a dataset with any name, but don't save the video under any group. The video should have a shape of (T, Lx, Ly), where T is the number of frames, and (Lx, Ly) is the lateral dimension of each frame. The support to more video formats will come soon. When doing format conversion, make sure the dimension is in the correct order. For example, if you save save the .h5 files from MATLAB, the shape of the dataset should be (Ly, Lx, T) in MATLAB. SeeInput filesfor more explanation of the dimension order.Use your own neuron masksSet the folder and file names of your mask files correctly. You need to change variabledir_maskto the folder containing the mask files.Currently, we only support .mat files as the neuron masks, so you need to convert the format of your neuron masks to .mat, and save the neuron masks in dataset 'FinalMasks'. The name of the masks file should beFinalMasks_{Exp_ID}.mat, where the{Exp_ID}is the name of the corresponding video. The neuron masks should be saved as a 3D array namedFinalMasks, and the dimension should be (Ly, Lx, n) in MATLAB, where Ly and Lx are the same as the lateral dimension of the video, and n is the number of neurons.The masks created by some manual labeling software may contain an empty (all zero) image as the first frame. You need to remove the empty frame before saving them.Set the unmixing parametersThe list of video names (e.g.,list_Exp_ID = ['c28_163_244'])The folder of the video (e.g.,dir_video='./data');The folder of the neuron masks (e.g.,dir_masks='./data');The folder of the output traces (e.g.,dir_traces='./data/unmixed_traces');list_alphais the list of tested alpha;multi_alphadetermines the option to deal with multiple elements inlist_alpha. False means the largest element providing non-trivial output traces will be used, which can be differnt for different neurons. True means each element will be tested and saved independently. These options are equivalent when there is only one element inlist_alpha;Qclipis clipping quantile. Traces lower than this quantile are clipped to this quantile value;th_pertminis maximum pertentage of unmixed traces equaling to the trace minimum;epsilonis the minimum value of the input traces after scaling and shifting;th_residualis the maximum factorization residual if this value is not zero;nbinis the temporal downsampling ratio;bin_optiondetermines the temporal downsampling option. It can be 'downsample', 'sum', or 'mean'. It is not used when nbin == 1;flexible_alphadetermines whether a flexible alpha strategy is used when the smallest alpha in "list_alpha" already caused over-regularization.Set alpha as a user-defined value or using cross-validationAmong the above parameters, we think most parameters do not need to be changed, but an optimizedlist_alphacan improve the unmixing accuracy. In our paper, we optimized the alpha using cross-validation, but it requires manual labeling of many traces, which is time consuming. Users can start withlist_alpha = [1], because we showed that most of our optimized alpha are close to 1, and using a user-defined initial alpha=1 can give nearly the same accuracy for most videos without extreme conditions.However, if the experimental conditions are very different from our test datasets, using cross-validation to optimize alpha will potentially be more reliable. Because cross-validation requires multiple videos, we don't provide demo for cross-validation, but users can run through our paper reproduction code in thepaper reproduction repoto see how cross-validation can be done. Here, we will briefly introduce how to perform cross-validation, and use our ABO dataset as an example. The following folders will refer to the folder in the paper reproduction repo.(1) Find a dataset containing multiple videos with similar experimental conditions. The three major datasets in our paper, ABO, NAOMi, 1p, are all qualified. (2) Manually label ground truth transients using the MATLAB GUI in the folderTemporalLabelingGUI(We already provided the ground truth transients for the paper reproduction datasets). (3) Run TUnCaT with multiple alpha inlist_alphausingTUnCaT_multi_ABO.py. For a new dataset, you can also start withdemo_TUnCaT.py, and setlist_alpha = [0.1, 0.2, 0.3, 0.5, 1, 2, 3, 5, 10]. Make suremulti_alpha = True. (4) Calculate the F1 scores of all videos with different alpha usingevaluation/eval_ABO_ours.m. (5) Find the optimal alpha for each cross-validation round usingevaluation/cross_validation.m. You can load the file storing the calculated F1 scores, or runcross_validation.mimmediately aftereval_ABO_ours.mto avoid reloading the F1 scores.CitationIf you use any part of this software in your work, please cite our paper:
Bao, Y., E. Redington, A. Agarwal, and Y. Gong, Decontaminate traces from fluorescence calcium imaging videos using targeted nonnegative matrix factorization. Frontiers in Neuroscience (2021 (in press)). doi:10.3389/fnins.2021.797421.Licensing and CopyrightTUnCaT is released underthe GNU License, Version 2.0.Sponsors