Package: competitionassay
Competition AssayIntroductionThis repository contains scripts to analyze competition assays. The scripts are used by the paper: ...TODOS:merge to main branchwrite the test files for the scriptslocal competition.py     => look at the histograms and see how we can define a paramter based on the data (e.g. the dominance parameter how dominant a certain species is in its area, or something like a "freedom" parameter that describes in how much area there is no competition expected)create an environment.yml file for: conda env create -f environment.ymlfind a better way to get the repetition name inside the script Area_competition.py at the moment it is very ugly:repetition_name=[WT_binary[i][-12:-4]foriinrange(len(WT_binary))]Area_competition: for b and c corresponding csv filesbiofilm identification: normalize with the intensity of a single cell, if segmentation is able to detect single cellslocal competition:subtract background intensity first (based on 2-class classification)as a better intensity parameter: compare ratios between mutant and WT intensity instead of the absolute (normalized) value as it is right now. Then compare these ratios with the ones from the distinct area (only X or only Y), to have a baselineTable of ContentsIntroductionInstallationHow to organize your data and the scriptsHow to run the programStep 1 - DenoisingGoogle Colab (recommended in case you should have problems with tensorflow on your machine)Step 2 - Segmentation/BinarizationStep 3 - QuantificationArea_competition.pyLocal_Competition.pyBiofilm_Identification.pyHow to train the model yourselfnoise2void resourcesRandomForestClassifierFAQInstallationFor Anaconda/Miniconda Users that want to run everything locally:create and activate an environment that has python 3.9condacreate-ncompetition_envpython=3.9
condaactivatecompetition_envInside the activated environment, type:pipinstallcompetitionassay[all]if you want to run noise2void on Colab and if you do not need napari for drawing GT yourself, then type:pipinstallcompetitionassaythe difference is, that the "all" version has n2v, tensorflow and napari installed.If this does not work, please take a look at theFAQsection.How to organize your data and the scriptstake a look at the structure in the "examples" folder. This is how your data should be organized. So the structure is, to have a base directory (e.g. example_data) and inside you have it organized like:-example_data
|----competitions
|    |   
|    |---competition_2_WTmScarlet_dwspFmNeonGreen
|    |   |
|    |   |--TW_growth
|    |      |   WT_name_repetition_1.tif
|    |      |   WT_name_repetition_2.tif
|    |      |   WT_name_repetition_3.tif
|    |      |   ...
|    |      |   Mutant_name_repetition_1.tif
|    |      |   Mutant_name_repetition_2.tif
|    |      |   Mutant_name_repetition_3.tif
|    |      |   ...
|    |---other_competition_folders
|    |   
|    |   |--TW_growth
|    |       |   ...
|    |
|----other foldersHow to run the programdownload the data, the models (noise2void and RandomForestClassifier) and the scripts from the repository. (or the complete repository)open your terminal and navigate to the directoryopen the scripts which you would like to test (e.g. start with Denoise_Imgs.py) and modify the base directory and the model path (if necessary)then run the scripts from your terminal or from your IDE (e.g. VSCode)conda activate competition_env
cd /path/to/your/CompetitionAssay

python3 scripts/Denoise_Imgs.py
python3 scripts/Denoising_and_Binarization.py
python3 scripts/...Step 1 - DenoisingThe first part is denoising usingnoise2void(GitHub,paper). This can be done on your machine with the Denoise_Imgs.py script, but also on Google Colab with the example script "noise2void_prediction.ipynb". The advantage of Google Colab is that you do not need to install tensorflow on your computer and you can use the GPU of Google Colab for the denoising. The disadvantage is that you need to upload your data to Google Drive and download it again after the denoising.For making the predictions, it is necessary to load the pretrained model. This can be found under the following link: TODO: [add here a link to the model - responsibility of the lab]The denoised files are automatically saved in a folder together with your files. The folder is called "denoised".Google Colab (recommended in case you should have problems with tensorflow on your machine)create a google drive account if you dont have it yetcreate a main folder and this file structure:- main folder
--- noise2void_prediction.ipynb
--- models/
    --- n2v_transwell
--- dataset_competition_assays/
    --- competition_2_WTmScarlet_dwspFmNeonGreen/
        --- inoculum/
        --- TW_growth/save your data theresave the model theresave the colab script (noise2void_prediction.ipynb) there - this script can be found in the repositoryrun the "noise2void_prediction.ipynb" script in google colab (it will create a folder called "denoised" (inside the TW_growth) in which all results are saved)now download the complete "dataset_competition_assays" to have it locally on your computer (or if you have it already on your computer, then download only the "denoised" folder and add it to the TW_growth folder)Step 2 - Segmentation/BinarizationThe denoised files are used for binarization via theRandom Forest Classifier. The RFC outputs 3 labels (1=background, 2=cell, 3=blurry regions), which will be reduced to a binary image (0=background, 1=cell). To run this script, open the"Denoising_and_Binarization.py"script and set the correct directories to the data and to the model (stored under"models/RandomForestClassifier_transwell/"). Also, if you have not done the denoising yet, set the parameter "perform_denoising" to True. In this case, please also direct to the "model_name" and "model_dir" of thenoise2void model.Once you run the script, a folder called "binary" will be created. This folder consist of all binary images that are produced by the RFC. If you want to visualize the results as image-mask overlay, then please use the scriptVisualize_RFC_Results.py. This script will create a folder called "RFC_output" in which all overlays are saved.Step 3 - QuantificationThere are different scripts for quantification. Not all of them ended up in the paper. The scripts areArea_competition.py,Local_Competition.py,Biofilm_Identification.pyArea_competition.pyThis script measures for every competition the covered areas by the WT and the competing Mutant and also the corresponding intensity. The output are several csv files and plots. Every competitionassay consist of several repetitions of the same competition (e.g. WT against wsp, Repetition 1, Repetition 2, ...). Every repetition (WT, Mutant) consist of several clusters (you can compare it with the results fromStep 2when using "Visualize_RFC_Results.py"). These clustered are labeled and every cluster results in an area and an intensity.These values are saved as csv file. The files have a name like "R2_002-1.csv". The corresponding area plot is called "name_competition_histogram.png".Finally, a summary of all repetitions is saved as csv file with the name "area_covered.csv" and the corresponding plot is called "area_covered.png".Local_Competition.pyThis script is now taking a closer look into the actual competitions. It is detecting the regions, in which a cluster has overlap with another cluster (competition) and in which regions the clusters are alone (no competition). The script returns one csv file and one figure per competition. The csv file contains of the following parameters:NameArea WT in Mutant (in percent)it describes how much area, in which the mutant spreads, is also covered by the WT. In this area, the Mutant competes with the WT. (1- Area WT in Mutant) is the percentage of area in which the Mutant is without competitionArea Mutant in WT (in percent)it is the same logic as aboveNorm_intensity_density_WT_in_Mutantit is a measure for the brightness of the clusters from the WT that is in the region, covered by the mutantthe brightness is normalized by approximately the maximum of the intensity from the WT (details indatahandling.py)this parameter can be used to compare the intensity of WT in the region of the Mutant with the intensity of WT in the region where the WT is alone (no competition)Norm_intensity_density_WT_alonethis parameter is the same as above, but it is the intensity of the WT in the region where the WT is alone (no competition, but also no protection from e.g. antibiotics)Norm_intensity_density_WT_generalit is a measure for the brightness of the clusters from the WT in general. This parameter is helpful to interpret the results of the normalized intensity, because one can compare the Norm_intensity_density_WT_in_Mutant and the Norm_intensity_density_WT_alone, with the Norm_intensity_density_WT_general to see, if the intensity increases or decreases for the WT, when being in the region of the mutant.The output can be read as:AreaFrom all the area in which the Mutant is colonized, it is sharing (co-colinized) 60 percent with the WT. The rest of the area (40 percent) is not colonized by the WT, so the mutant is alone there (no competition)From all the area in which the WT is colonized, it is sharing (co-colinized) 30 percent with the Mutant. The rest of the area (70 percent) is not colonized by the Mutant, so the WT is alone there (no competition)Intensitythe normalized intensity of the WT in general is 0.4, so in average it is 40 percent of the highest intensities of the WT (I did not normalize with the maximum intensity, since it is not resistant to outliers)the normalized intensity of the WT in the region of the Mutant is 0.45, so a bit higher than the averagethe normalized intensity of the WT in the region where the WT is alone is 0.4, so a bit lower than average and lower than the area with overlap.the same principal applies for the intensities of the mutantSo for example, if you want to test, if the WT is more resistant to antibiotics in the region of the Mutant, then you can compare the normalized intensity of the WT in the region of the Mutant with the normalized intensity of the WT in the region where the WT is alone. If the normalized intensity of the WT in the region of the Mutant is higher than the normalized intensity of the WT in the region where the WT is alone, then the WT is more resistant to antibiotics in the region of the Mutant (e.g. because it is protected by the biofilm).Biofilm_Identification.pyThis script is not used in the paper, but it might help to define biofilms based on the cluster size and cluster intensity. Results are saved in "Competition_results/" and they are scatter plots that are splitted into 4 regions, based on brightness and cluster size.How to train the model yourselfnoise2void resourcesthis algorithm is non-supervised and does not need a ground truth for trainingtake a look at theGitHuborpapertake a look atDigitalSreeni YoutubeRandomForestClassifierfirst make a GT, to do so, take a look atRFC_Draw_GT.ipynbwhich needsnaparito be installedthen train the network withTrain_RFC.pythe model can be saved wherever you want, just make sure to change the path in theDenoising_and_Binarization.pyscript when you want to use your own model.there is example trainingsdata available atexample_data/RFC_Trainingsdata/FAQQ:I get an error when I try to install the package with pip install competitionassayTry to set up an empty conda environment with python 3.9. Then instead of using the pip install command, download the requirements_no_DL.txt file from the repository and type:condaactivatecompetition_env
condainstallcompetitionassayThen continue running the scripts until they lead to errors like: No module named ...Install these modules with:pipinstalltensorflow=2.12.0
pipinstalln2v==0.3.2
pipinstallnapari[all]AuthorEric Schmidt