Repository: maximilianh/cellBrowser (language: python)

Issue:
merge back from new repo.

Fixed a typo
This fixes an issue I noticed when noticed that expression in cellbrowser did not match the output of FeaturePlot of Seurat. 
In my case, `exprArr` had `int64` type, it was not converted to `uint32` and consequently the encoded string `exprStr` was twice as long. But only the first half was received in JS by `cbData.js::gunzipAndConvert`, leading to many zeros in the decoded array.

I assume this is a typo.
Error in in CreateSeuratObject when running cbSeurat on an expression matrix
Hi, 
I'm trying to run cbSeurat on an expression matrix and keep getting this error on in CreateSeuratObject(..). 
I downloaded your example from adultPancreas matrix and tried running it and get the same error than when running my matrix. 


adultPancreas% ls -l 
total 22396
-rw-r--r--. 1 root root 22707068 Feb 27  2019 exprMatrix.tsv.gz
-rw-r--r--. 1 root root   522326 Apr  2  2020 meta.tsv
adultPancreas% cbSeurat --exprMatrix=exprMatrix.tsv.gz --name=myDataset --outDir=seurat-out
WARNING:root:seurat.conf not found, using default values for all Seurat options. Use --init to create a sample file.
INFO:root:Creating seurat-out
INFO:root:Wrote R script to seurat-out/runSeurat.R
INFO:root:running seurat-out/runSeurat.R through Rscript
Loading required package: R.methodsS3
R.methodsS3 v1.8.1 (2020-08-26 16:20:06 UTC) successfully loaded. See ?R.methodsS3 for help.
R.oo v1.24.0 (2020-08-26 16:11:58 UTC) successfully loaded. See ?R.oo for help.
R.utils v2.11.0 (2021-09-26 08:30:02 UTC) successfully loaded. See ?R.utils for help.
R version 4.0.5 (2021-03-31)
Seurat loaded, version 4.1.0
[1] "Seurat: Reading data"
[1] "Seurat: Reading data"
[1] "Loading input data matrix"
NULL
[1] "Loading tsv file with read.delim"
[1] "Seurat: Setup"
Error in CreateSeuratObject(raw.data = mat, min.cells = 3, min.genes = 200) : 
  argument "counts" is missing, with no default
Execution halted

real	0m51.316s
user	0m44.770s
sys	0m3.390s
INFO:root:Wrote logfile of R run to seurat-out/analysisLog.txt
ERROR:root:R script did not complete successfully. Check seurat-out/runSeurat.R and analysisLog.txt.
python >= 3.9 compatibility (tostring -> tobytes)
The following error occurs when calling `cbBuild` with python v.3.10:
>INFO:root:Auto-detect: Numbers in matrix are of type 'float'
ERROR:root:Unexpected error: (<class 'AttributeError'>, AttributeError("'array.array' object has no attribute 'tostring'"), <traceback object at 0x103f1ffc0>)
Traceback (most recent call last):
  File "/Users/kriemo/.local/pipx/venvs/cellbrowser/lib/python3.10/site-packages/cellbrowser/cellbrowser.py", line 4783, in cbBuildCli
    build(confFnames, outDir, port, redo=options.redo)
  File "/Users/kriemo/.local/pipx/venvs/cellbrowser/lib/python3.10/site-packages/cellbrowser/cellbrowser.py", line 4598, in build
    convertDataset(inDir, inConf, outConf, datasetDir, redo)
  File "/Users/kriemo/.local/pipx/venvs/cellbrowser/lib/python3.10/site-packages/cellbrowser/cellbrowser.py", line 3955, in convertDataset
    convertExprMatrix(inConf, outMatrixFname, outConf, sampleNames, geneToSym, datasetDir, needFilterMatrix)
  File "/Users/kriemo/.local/pipx/venvs/cellbrowser/lib/python3.10/site-packages/cellbrowser/cellbrowser.py", line 3297, in convertExprMatrix
    matType = matrixToBin(outMatrixFname, geneToSym, binMat, binMatIndex, discretBinMat, discretMatrixIndex, metaSampleNames, matType=matType, genesAreRanges=genesAreRanges)
  File "/Users/kriemo/.local/pipx/venvs/cellbrowser/lib/python3.10/site-packages/cellbrowser/cellbrowser.py", line 2052, in matrixToBin
    exprStr, minVal = exprEncode(geneId, exprArr, matType)
  File "/Users/kriemo/.local/pipx/venvs/cellbrowser/lib/python3.10/site-packages/cellbrowser/cellbrowser.py", line 1934, in exprEncode
    exprStr = array.array(arrType, exprArr).tostring()
AttributeError: 'array.array' object has no attribute 'tostring'
>

The `tostring()` method for `array.array` objects has been removed in python >= v3.9. This PR substitutes in the alias method `tobytes()`, which fixes the error, and is backwards compatible with python >= v3.2. 

https://docs.python.org/3.9/whatsnew/3.9.html#removed


Discussion / hints:

Fix patch:
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
index f70686d7..4f36d0c6 100644
--- a/CODE_OF_CONDUCT.md
+++ b/CODE_OF_CONDUCT.md
@@ -1,9 +1,7 @@
 # Contributor Code of Conduct
 
-Please be nice, respectful of differing viewpoints and experiences and acknowledge that every piece of code has a history that is 
-not free of time/money/personal constraints.
+Please be nice, respectful of differing viewpoints and experiences. Note that every piece of code has a history that is not free of time/money/personal constraints.
 
-What sometimes comes up is the coding style and libraries used. The project does not use every popular Javascript framework. This has the disadvantage that some things are slower to implement at first, but has the advantage that we potentially can get contributions from programmers who are not yet on the most recent frameworks and also that when something breaks, we do not have to dig through the implicit behavior that is part of the framework. We hope that contributors are not put off by this. We do appreciate any ideas on how to improve the UI system, possibly by adding a UI framework in a light-weight way.
+You may wonder about the coding style and libraries used. The project does not use every popular Javascript framework. This has the disadvantage that some things are slower to implement at first, but has the advantage that we potentially can get contributions from programmers who are not yet on the most recent frameworks and also that when something breaks, we do not have to dig through the implicit behavior that is part of the framework. We hope that contributors are not put off by this. We do appreciate any ideas on how to improve the UI system, possibly by adding a UI framework in a lightweight way or just in any way, as our experience with Javascript frameworks is extremely limited.
 
-This is a relatively small software project and all contributors have all been extremely nice, 
-which is why this Code of Conduct is very short. 
+This is a relatively small software project and all contributors have all been extremely nice, which is why this Code of Conduct is very short. 
diff --git a/README.rst b/README.rst
index e6df367b..df760736 100644
--- a/README.rst
+++ b/README.rst
@@ -30,11 +30,12 @@ Many labs host their data at cells.ucsc.edu by sending it to us, but some groups
 * Accelerating Medicine Partnership Consortium, https://immunogenomics.io/cellbrowser/, used in `Zhang et al. 2018 <https://www.biorxiv.org/content/10.1101/351130v1>`_ and `Der et al 2018 <https://www.biorxiv.org/content/10.1101/382846v1>`_
 * Sansom Lab, Oxford, https://sansomlab.github.io for `Croft et al, Nature 2019 <https://www.nature.com/articles/s41586-019-1263-7>`_
 * http://caire.ipmc.cnrs.fr/cellbrowser/Differentiation/ (URL has changed, contacted authors, no reply, new URL seems to be https://www.genomique.eu/cellbrowser/HCA/) Zaragosi group at IPMC CNRS Nice, for the manuscript https://dev.biologists.org/content/early/2019/09/25/dev.177428.abstract
-* Bin Ren lab, CAAtlas http://catlas.org/mousebrain/#!/home
+* Bing Ren lab, CATlas http://catlas.org/mousebrain/#!/home
 * Conrad lab at Charite Berlin: http://singlecell.charite.de/
 * `STAB: a spatio-temporal cell atlas of the human brain <https://stab.comp-sysbio.org/tool/cellbrowser/index.html>`_ from  `Song et al NAR 2021 <https://academic.oup.com/nar/article/49/D1/D1029/5911746>`_.
 * UCLA: http://mergeomics.research.idre.ucla.edu/PVDSingleCell/
 * Lako Lab at Newcastle University, UK: http://retinalstemcellresearch.co.uk/CorneaCellAtlas/ from `Collins et al. 2021. The Ocular Surface. <https://www.sciencedirect.com/science/article/pii/S1542012421000215>`_
+* RNA Bioscience Initiative: https://www.pneuroonccellatlas.org/ and https://github.com/rnabioco/lung-scrna
 
 These papers have cell browsers made at UCSC:
 
diff --git a/build/buildEnv.sh b/build/buildEnv.sh
index cee12581..cc1f0dd3 100644
--- a/build/buildEnv.sh
+++ b/build/buildEnv.sh
@@ -1,6 +1,6 @@
-export CBVERSION=v1.1.1
-export CBDATE=2021-09-01
-export CBLASTVERSION=v1.1.0
-export CBLASTDATE=2021-08-18
+export CBVERSION=v1.2.0
+export CBDATE=2022-05-13
+export CBLASTVERSION=v1.1.1
+export CBLASTDATE=2021-09-01
 #export CBLASTVERSION=v0.7.9 
 #export CBLASTDATE=2021-07-03
diff --git a/build/doPush.sh b/build/doPush.sh
index 68c0610d..d4c09e52 100755
--- a/build/doPush.sh
+++ b/build/doPush.sh
@@ -5,4 +5,7 @@ git merge develop
 git push
 git tag $CBVERSION
 git push origin $CBVERSION
+
+# this assumes that build is a child of the top level directory
+cd ..
 make pip
diff --git a/docs/howto.rst b/docs/howto.rst
index 12fb4d97..7f890c7d 100644
--- a/docs/howto.rst
+++ b/docs/howto.rst
@@ -30,6 +30,10 @@ Lastly, go into the output directory specified in the cbImportSeurat command and
   cd myRdsImport
   cbBuild -o ~/public_html/cb
 
+Or, if you don't have a webserver already, use the built-in one:
+
+  cbBuild -o /myHtmlFiles -p 8888
+
 You should now be able to access your cell browser from the web.
 
 How to use the cell browser export function in Seurat3
@@ -62,6 +66,10 @@ Next, go into the output directory specified in the cbImportSeurat command and r
   cd seurat-out
   cbBuild -o ~/public_html/cb
 
+Or, if you don't have a webserver already, start the built-in one:
+
+  cbBuild -o /myHtmlFiles -p 8888
+
 
 How to configue a basic cbSeurat pipeline
 ^^^^
@@ -103,11 +111,14 @@ Then, go into the output directory specified in the cbImportSeurat command and r
   cd scanpy-import
   cbBuild -o ~/public_html/cb
 
-You should now be able to access your cell browser from the web.
+Or, if you don't have a webserver already, start the built-in one:
 
+  cbBuild -o /myHtmlFiles -p 8888
 
+You should now be able to access your cell browser from the web or your local computer.
 
-How to convert a Scanpy object wihthin Python
+
+How to convert a Scanpy object within Python
 ^^^^
 
 It a few simple commands to build a ``cellbrowser.conf`` and all the files you need for a cell
@@ -163,7 +174,6 @@ Next, go into the output directory specified in the ``cbScanpy`` command and bui
   cd scanpy-out
   cbBuild -o ~/public_html/cb
 
-
 How to configue a basic cbScanpy pipeline
 ^^^^
 
@@ -361,7 +371,12 @@ You will specifically need to edit these lines to point to the flies that you ex
   
 You will still need to set the other `required settings <https://github.com/maximilianh/cellBrowser/blob/master/src/cbPyLib/cellbrowser/sampleConfig/cellbrowser.conf#L1>`_ in your cellbrowser.conf as well
 
+How to start the webserver without building datasets
+^^^^
 
+If you have stopped the built-in webserver and want to start it again, without rebuilding the entire dataset, use the cbUpgrade tool:
+
+  cbUpgrade -o /myHtmlFiles -p 8888
 
 How to visualize single-cell ATAC-seq data in the Cell Browser
 ^^^^
@@ -372,6 +387,7 @@ Step 1: Gather required files
 """"
 
 You will the following three files:
+
 * Expression matrix with cell names as columns and peak ranges as rows. 
 * Cell annotations/metadata
 * Layout coordinats (e.g. UMAP)
@@ -379,13 +395,13 @@ You will the following three files:
 Step 2: Determine GENCODE Gene Model version (optional)
 """"
 
-If you don't know the GENCODE version used, cbGenes can determine the most likely version used:
+If you don't know the GENCODE version used, cbGenes can determine the most likely version used from a list of gene identifiers:
 
 ::
 
  cbGenes guess exprMatrix.tsv.gz human
 
-The first column of this file should be gene symbols of GENCODE gene IDs.
+The first column of this file should be the gene symbols of GENCODE gene IDs (e.g. ENSG00000184779).
 
 Step 3: Download the gene model files
 """"
diff --git a/docs/images/favicon/android-chrome-192x192.png b/docs/images/favicon/android-chrome-192x192.png
new file mode 100644
index 00000000..d226d1cc
Binary files /dev/null and b/docs/images/favicon/android-chrome-192x192.png differ
diff --git a/docs/images/favicon/android-chrome-512x512.png b/docs/images/favicon/android-chrome-512x512.png
new file mode 100644
index 00000000..cd8edc10
Binary files /dev/null and b/docs/images/favicon/android-chrome-512x512.png differ
diff --git a/docs/images/favicon/apple-touch-icon.png b/docs/images/favicon/apple-touch-icon.png
new file mode 100644
index 00000000..73725691
Binary files /dev/null and b/docs/images/favicon/apple-touch-icon.png differ
diff --git a/docs/images/favicon/cellBrowserLogoRaw.png b/docs/images/favicon/cellBrowserLogoRaw.png
new file mode 100644
index 00000000..0e2e11f0
Binary files /dev/null and b/docs/images/favicon/cellBrowserLogoRaw.png differ
diff --git a/docs/images/favicon/favicon-16x16.png b/docs/images/favicon/favicon-16x16.png
new file mode 100644
index 00000000..fcb7c423
Binary files /dev/null and b/docs/images/favicon/favicon-16x16.png differ
diff --git a/docs/images/favicon/favicon-32x32.png b/docs/images/favicon/favicon-32x32.png
new file mode 100644
index 00000000..8690771b
Binary files /dev/null and b/docs/images/favicon/favicon-32x32.png differ
diff --git a/docs/images/favicon/favicon.ico b/docs/images/favicon/favicon.ico
new file mode 100644
index 00000000..9a5639c0
Binary files /dev/null and b/docs/images/favicon/favicon.ico differ
diff --git a/docs/quick_start.rst b/docs/quick_start.rst
index 2d9a1938..ac5a812b 100644
--- a/docs/quick_start.rst
+++ b/docs/quick_start.rst
@@ -15,7 +15,7 @@ Step 1: Download the example dataset
 
 Download the data using curl::
 
-  curl -ks https://cells.ucsc.edu/downloads/samples/mini.tgz 
+  curl -kO https://cells.ucsc.edu/downloads/samples/mini.tgz 
 
 This file contains all the pieces necessary to create a simple cell browser. 
 
@@ -24,7 +24,7 @@ Step 2: Unpack the dataset
 
 Next, we need to take this file and unpack it's contents::
 
-  tar xvz mini.tgz
+  tar -xvzf mini.tgz
 
 This will create a new ``mini`` directory contain the cell browser files. 
 
diff --git a/docs/submission.rst b/docs/submission.rst
index 68c92e4e..23fa437f 100644
--- a/docs/submission.rst
+++ b/docs/submission.rst
@@ -51,7 +51,7 @@ Dataset description
 """"
 
 Alongside your submission, it would be great if you filled out a 
-`desc.conf <https://cellbrowser.readthedocs.io/dataDesc.html>`_ file. At
+`desc.conf <https://cellbrowser.readthedocs.io/en/master/dataDesc.html>`_ file. At
 the very least, it should have the abstract, methods, and title filled out. 
 However, you are welcome to fill out more fields and make it as complete as 
 you would like. You can run ``cbBuild --init`` to copy an example desc.conf
diff --git a/src/cbPyLib/cellbrowser/R/cellbrowser.R b/src/cbPyLib/cellbrowser/R/cellbrowser.R
index 5d32d16d..f4c725e2 100644
--- a/src/cbPyLib/cellbrowser/R/cellbrowser.R
+++ b/src/cbPyLib/cellbrowser/R/cellbrowser.R
@@ -64,8 +64,9 @@ findMatrices = function(object, slotNames ) {
       if (slotName == "scale.data")
           slotName <- "scale" #  dots in filenames are not good
       # do not use any prefixes if we export just a single matrix (stay compatible with old code)
+      # empty string didn't work with R names(), so using special value "###" now
       if (length(slotNames)==1)
-          slotName <- ""
+         slotName <- "###"
       slotMatrices [[slotName]] <- mat
   }
   return(slotMatrices)
@@ -83,9 +84,12 @@ saveMatrix <- function(counts, dir, prefix, use.mtx) {
   # Export expression matrix
   message("Writing matrix with prefix ",prefix," to directory ",dir, "(use.mtx is ", use.mtx, ")")
   too.big = ((((ncol(counts)/1000)*(nrow(counts)/1000))>2000) && is(counts, 'sparseMatrix'))
-  if (prefix!="" && !endsWith(prefix, "_"))
+  if (prefix=="###")
+      prefix <- ""
+  else if (prefix!="" && !endsWith(prefix, "_"))
       prefix <- paste0(prefix, "_")
 
+
   if (use.mtx || too.big) {
         # we have to write the matrix to an mtx file
         matrixPath <- file.path(dir, paste(prefix, "matrix.mtx", sep=""))
@@ -408,6 +412,9 @@ ExportToCellbrowser <- function(
   }
 
   firstPrefix <- names(slotMatrices)[1]
+  if (firstPrefix=="###")
+      firstPrefix = ""
+
   if (length(slotMatrices)==1)
       matSep = ""
   else
@@ -420,10 +427,7 @@ ExportToCellbrowser <- function(
   matrixLabels[matrixLabels=="scale.data"] <- "scaled"
   matrices.conf <- sprintf(" {'label':'%s','fileName':'%s_exprMatrix.tsv.gz'}", names(slotMatrices), names(slotMatrices))
 
-  if (length(slotMatrices)==1)
-     matrices.string <- ""
-  else
-     matrices.string <- paste0("matrices=[", paste(matrices.conf, collapse = ",\n"), "]" )
+  matrices.string <- paste0("matrices=[", paste(matrices.conf, collapse = ",\n"), "]" )
 
   matrixOutPath <- sprintf("%s%sexprMatrix.tsv.gz", firstPrefix, matSep)
   if (use.mtx) {
diff --git a/src/cbPyLib/cellbrowser/cbWeb/css/cellBrowser.css b/src/cbPyLib/cellbrowser/cbWeb/css/cellBrowser.css
index 7aa5ee25..d354728f 100644
--- a/src/cbPyLib/cellbrowser/cbWeb/css/cellBrowser.css
+++ b/src/cbPyLib/cellbrowser/cbWeb/css/cellBrowser.css
@@ -90,11 +90,35 @@ section { font-weight: bold; padding-top: 5px; padding-bottom: 5px }
     font-weight: bold;
 }
 
-.context-menu-item { font-size: 13px; padding: 3px }
-.contextmenu-customwidth { width: 200px !important; min-width: 200px !important}
-.tpGeneBarCell { font-size: 12px; width:58px; color: black; padding-left:1px; padding-right:1px; border: 1px solid #AAA; max-width: 80px; background: #DDDDDD; color:#AAAAAA }
+.context-menu-item { 
+        font-size: 13px; 
+        padding: 3px 
+}
+
+.contextmenu-customwidth { 
+        width: 200px !important; 
+        min-width: 200px !important
+}
+
+.tpGeneBarCell { 
+        font-size: 12px; 
+        color: black; 
+        padding-left:1px; 
+        padding-right:1px; 
+        /* border: 1px solid #AAA; */
+        background: #DDDDDD; 
+        color:#AAAAAA;
+        display: inline-block;
+}
+
 .tpGeneBarCellSelected { outline: 2px solid black}
-#tpGeneTable { table-layout: fixed; background-color: #EEE; white-space: nowrap; border-collapse:collapse }
+
+#tpGeneTable { 
+        table-layout: fixed; 
+        background-color: #EEE; 
+        white-space: nowrap; 
+        border-collapse:collapse 
+}
 
 #tpPeakListTitle { 
         padding-top: 8px;
@@ -168,6 +192,7 @@ input[type=checkbox], input[type=radio] {
         padding:0;
         padding-top: 5px;
         height: 100%;
+        overflow-y: scroll;
 }
 
 .tpMetaBox { font-size: 13px; cursor:default; overflow:hidden }
diff --git a/src/cbPyLib/cellbrowser/cbWeb/ext/chosen-sprite@2x.png b/src/cbPyLib/cellbrowser/cbWeb/ext/chosen-sprite@2x.png
new file mode 100644
index 00000000..2a90c46c
Binary files /dev/null and b/src/cbPyLib/cellbrowser/cbWeb/ext/chosen-sprite@2x.png differ
diff --git a/src/cbPyLib/cellbrowser/cbWeb/js/cbData.js b/src/cbPyLib/cellbrowser/cbWeb/js/cbData.js
index 4035f996..633c263e 100644
--- a/src/cbPyLib/cellbrowser/cbWeb/js/cbData.js
+++ b/src/cbPyLib/cellbrowser/cbWeb/js/cbData.js
@@ -120,7 +120,10 @@ var cbUtil = (function () {
 
         oReq.onload = cbUtil.onDoneBinaryData;
         oReq.onprogress = onProgress;
-        oReq.onerror = function(e) { alert("Could not load file "+url); };
+        oReq.onerror = function(e) { 
+            // Github rejects accept-encoding: headers coming from Firefox at the moment.
+            alert("Could not load file "+url+". If this is Firefox and running on Github, please contact us."); 
+        };
         oReq._onDone = onDone; // keep this for the callback
         oReq._otherInfo = otherInfo; // keep this for the callback
         oReq._arrType = arrType; // keep this for the callback casting
@@ -405,9 +408,13 @@ function CbDbFile(url) {
            dsUrl = dsUrl+"?"+md5;
         cbUtil.loadJson(dsUrl, function(data) { self.conf = data; gotOneFile();});
 
-        // start loading gene offsets in background, because this takes a while
-        var osUrl = cbUtil.joinPaths([this.url, "exprMatrix.json"]);
-        cbUtil.loadJson(osUrl, function(data) { matrixIndex = data; gotOneFile();}, true);
+        if (self.name!='') {
+            // start loading gene offsets in the background now, because this takes a while
+            var osUrl = cbUtil.joinPaths([this.url, "exprMatrix.json"]);
+            cbUtil.loadJson(osUrl, function(data) { matrixIndex = data; gotOneFile();}, true);
+        } else {
+            gotOneFile();
+        }
     };
 
     this.loadCoords = function(coordIdx, onDone, onProgress) {
@@ -721,7 +728,8 @@ function CbDbFile(url) {
         /* for both gene and ATAC mode: */
         /* return an array [start, end, name] given a locus description (=a string: gene symbol or chr|start|end) */
         var off = null;
-        if (name.includes("|")) { // atac mode: name is chrom|start|end
+        //if (name.includes("|")) { // atac mode: name is chrom|start|end
+        if (self.peakOffsets !== null) { // atac mode: name is chrom|start|end
             let pos = name.split("|");
             off = self.findAtacOffsets(pos[0], parseInt(pos[1]), parseInt(pos[2]));
         }
@@ -1457,14 +1465,16 @@ function CbDbFile(url) {
        var loadCounter = 0;
        if (geneSyms) {
            for (var i=0; i<geneSyms.length; i++) {
-               var sym = geneSyms[i][0];
-               if (! (sym in validGenes)) {
-                  alert("Error: "+sym+" is in quick genes list but is not a valid gene");
-                  continue;
-               }
+               var geneId = geneSyms[i][0];
+               //if (! (sym in validGenes)) {
+                  //alert("Error: "+sym+" is in quick genes list but is not a valid gene");
+                  //continue;
+               //}
+               if (geneId.indexOf("|")!==-1)
+                   geneId = geneId.split("|")[0];
 
                 self.loadExprAndDiscretize(
-                   sym,
+                   geneId,
                    function(exprVec, discExprVec, geneSym, geneDesc, binInfo) {
                        self.quickExpr[geneSym] = [discExprVec, geneDesc, binInfo];
                        loadCounter++;
diff --git a/src/cbPyLib/cellbrowser/cbWeb/js/cellBrowser.js b/src/cbPyLib/cellbrowser/cbWeb/js/cellBrowser.js
index 25097ab9..bbabf15b 100644
--- a/src/cbPyLib/cellbrowser/cbWeb/js/cellBrowser.js
+++ b/src/cbPyLib/cellbrowser/cbWeb/js/cellBrowser.js
@@ -142,6 +142,86 @@ var cellbrowser = function() {
             return str;
         }
 
+    // Median of medians: https://en.wikipedia.org/wiki/Median_of_medians
+    // find median in an unsorted array, worst-case complexity O(n).
+    // from https://gist.github.com/wlchn/ee15de1da59b8d6981a400eee4376ea4
+    const selectMedian = (arr, compare) => {
+        return _selectK(arr, Math.floor(arr.length / 2), compare);
+    };
+
+    const _selectK = (arr, k, compare) => {
+        if (!Array.isArray(arr) || arr.length === 0 || arr.length - 1 < k) {
+            return;
+        }
+        if (arr.length === 1) {
+            return arr[0];
+        }
+        let idx = _selectIdx(arr, 0, arr.length - 1, k, compare || _defaultCompare);
+        return arr[idx];
+    };
+
+    const _partition = (arr, left, right, pivot, compare) => {
+        let temp = arr[pivot];
+        arr[pivot] = arr[right];
+        arr[right] = temp;
+        let track = left;
+        for (let i = left; i < right; i++) {
+            // if (arr[i] < arr[right]) {
+            if (compare(arr[i], arr[right]) === -1) {
+                let t = arr[i];
+                arr[i] = arr[track];
+                arr[track] = t;
+                track++;
+            }
+        }
+        temp = arr[track];
+        arr[track] = arr[right];
+        arr[right] = temp;
+        return track;
+    };
+
+    const _selectIdx = (arr, left, right, k, compare) => {
+        if (left === right) {
+            return left;
+        }
+        let dest = left + k;
+        while (true) {
+            let pivotIndex =
+                right - left + 1 <= 5
+                ? Math.floor(Math.random() * (right - left + 1)) + left
+                : _medianOfMedians(arr, left, right, compare);
+            pivotIndex = _partition(arr, left, right, pivotIndex, compare);
+            if (pivotIndex === dest) {
+                return pivotIndex;
+            } else if (pivotIndex < dest) {
+                left = pivotIndex + 1;
+            } else {
+                right = pivotIndex - 1;
+            }
+        }
+    };
+
+    const _medianOfMedians = (arr, left, right, compare) => {
+        let numMedians = Math.ceil((right - left) / 5);
+        for (let i = 0; i < numMedians; i++) {
+        let subLeft = left + i * 5;
+        let subRight = subLeft + 4;
+        if (subRight > right) {
+            subRight = right;
+        }
+        let medianIdx = _selectIdx(arr, subLeft, subRight, Math.floor((subRight - subLeft) / 2), compare);
+        let temp = arr[medianIdx];
+        arr[medianIdx] = arr[left + i];
+        arr[left + i] = temp;
+        }
+        return _selectIdx(arr, left, left + numMedians - 1, Math.floor(numMedians / 2), compare);
+    };
+
+    const _defaultCompare = (a, b) => {
+        return a < b ? -1 : a > b ? 1 : 0;
+    };
+    // End median of medians
+
     function debug(msg, args) {
         if (DEBUG) {
             console.log(formatString(msg, args));
@@ -322,6 +402,9 @@ var cellbrowser = function() {
 
     function iWantHue(n) {
         /* a palette as downloaded from iwanthue.com - not sure if this is better. Ellen likes it */
+        if (n>30)
+            return null;
+
         var colList = ["7e4401", "244acd", "afc300", "a144cb", "00a13e",
             "f064e5", "478700", "727eff", "9ed671", "b6006c", "5fdd90", "f8384b",
             "00b199", "bb000f", "0052a3", "fcba56", "005989", "c57000", "7a3a78",
@@ -443,8 +526,6 @@ var cellbrowser = function() {
         //var md5 = datasetInfo.md5;
         // the UCSC apache serves latin1, so we force it back to utf8
         gOpenDataset = datasetInfo; // for click handlers in the right panel
-        var thumbUrl = cbUtil.joinPaths([datasetInfo.name, "thumb.png"]);
-        preloadImage(thumbUrl); // many datasets have thumb.png, so preload it now
 
         $.ajaxSetup({
             'beforeSend' : function(xhr) {
@@ -507,6 +588,7 @@ var cellbrowser = function() {
         "doi" : "Publication Fulltext",
         "arrayexpress" : "ArrayExpress",
         "ena_project" : "European Nucleotide Archive",
+        "hca_dcp" : "Human Cell Atlas DCP",
         "cirm_dataset" : "California Institute of Regenerative Medicine Dataset",
     };
 
@@ -522,6 +604,7 @@ var cellbrowser = function() {
         "ena_project" : "https://www.ebi.ac.uk/ena/data/view/",
         "cirm_dataset" : "https://cirm.ucsc.edu/d/",
         "arrayexpress" : "https://www.ebi.ac.uk/arrayexpress/experiments/",
+        "hca_dcp" : "https://data.humancellatlas.org/explore/projects/",
     }
 
     function htmlAddLink(htmls, desc, key, linkLabel) {
@@ -571,6 +654,24 @@ var cellbrowser = function() {
         htmls.push("<br>");
     }
 
+    function buildLinkToMatrix(htmls, dsName, matFname, label) {
+        /* link to a matrix file for download, handles mtx.gz */
+        if (label)
+            htmls.push("<b> Matrix for "+label+":</b> ");
+
+        htmls.push("<a href='"+dsName+"/"+matFname+"'>"+matFname+"</a>");
+        if (matFname.endsWith(".mtx.gz")) {
+            var prefix = "";
+            if (matFname.indexOf("_")!==-1)
+                prefix = matFname.split("_")[0]+"_";
+            var ftName = prefix+"features.tsv.gz";
+            htmls.push(", <a href='"+dsName+"/"+ftName+"'>"+ftName+"</a>");
+            var barName = prefix+"barcodes.tsv.gz";
+            htmls.push(", <a href='"+dsName+"/"+barName+"'>"+barName+"</a>");
+        }
+        htmls.push("<br>");
+    }
+
     function buildDownloadsPane(datasetInfo, desc) {
         var htmls = [];
         if (datasetInfo.name==="") { // the top-level desc page has no methods/downloads, it's just a dataset list
@@ -585,23 +686,28 @@ var cellbrowser = function() {
                 htmls.push("The downloads section has been deactivated by the authors.");
                 htmls.push("Please contact the dataset authors to get access.");
             } else {
-                if (desc.matrixFile!==undefined && desc.matrixFile.endsWith(".mtx.gz")) {
-                    htmls.push("<p><b>Expression in MTX format:</b> <a href='"+datasetInfo.name);
-                    htmls.push("/matrix.mtx.gz'>matrix.mtx.gz</a>");
-                    htmls.push(", <a href='"+datasetInfo.name);
-                    htmls.push("/features.tsv.gz'>features.tsv.gz</a>");
-                    htmls.push(", <a href='"+datasetInfo.name);
-                    htmls.push("/barcodes.tsv.gz'>barcodes.tsv.gz</a>");
+                if (desc.matrices) {
+                    htmls.push("<p>");
+                    for (var key in desc.matrices) {
+                        var mat = desc.matrices[key];
+                        buildLinkToMatrix(htmls, datasetInfo.name, mat.fileName, mat.label);
+                    }
+                    htmls.push("</p>");
+                } else if (desc.matrixFile!==undefined && desc.matrixFile.endsWith(".mtx.gz")) {
+                    htmls.push("<p>");
+                    var matBaseName = desc.matrixFile.split("/").pop();
+                    buildLinkToMatrix(htmls, datasetInfo.name, matBaseName, "dataset");
+                    htmls.push("</p>");
                 } else {
-                htmls.push("<p><b>Expression matrix:</b> <a href='"+datasetInfo.name);
-                htmls.push("/exprMatrix.tsv.gz'>exprMatrix.tsv.gz</a>");
+                    htmls.push("<p><b>Matrix:</b> <a href='"+datasetInfo.name);
+                    htmls.push("/exprMatrix.tsv.gz'>exprMatrix.tsv.gz</a>");
                 }
                 if (desc.unitDesc)
-                    htmls.push("<br>Values are: "+desc.unitDesc);
+                    htmls.push("<br>Values in matrix are: "+desc.unitDesc);
                 htmls.push("</p>");
 
                 if (desc.rawMatrixFile) {
-                    htmls.push("<p><b>Raw expression matrix:</b> <a href='"+datasetInfo.name);
+                    htmls.push("<p><b>Raw count matrix:</b> <a href='"+datasetInfo.name);
                     htmls.push("/"+desc.rawMatrixFile+"'>"+desc.rawMatrixFile+"</a>");
                     if (desc.rawMatrixNote)
                         htmls.push("<br>"+desc.rawMatrixNote);
@@ -860,6 +966,7 @@ var cellbrowser = function() {
         let topName = datasetInfo.name.split("/")[0];
         if (pageAtUcsc()) {
             if (datasetInfo.name!=="") {
+                // Only do this if this is not the root dataset
                 if ((datasetInfo.parents) && (datasetInfo.parents.length > 1)) {
                     // if the dataset is a collection
                     htmls.push("<b>Direct link to this collection for manuscripts: </b> https://"+topName+".cells.ucsc.edu");
@@ -869,9 +976,18 @@ var cellbrowser = function() {
                     htmls.push("<b>Direct link to this plot for manuscripts: </b> https://"+topName+".cells.ucsc.edu");
                     htmls.push("<br>");
                 }
+
+            console.log(datasetInfo);
+
+            if ( datasetInfo.atacSearch) {
+                    htmls.push("<b>ATAC-seq search gene models: </b>" + datasetInfo.atacSearch);
+                    htmls.push("<br>");
+            }
+
+            htmls.push("<p style='padding-top: 15px'><small>Cell Browser dataset ID: "+datasetInfo.name+
+                    "</small></p>");
             }
         }
-        htmls.push("<p style='padding-top: 15px'><small>Cell Browser dataset ID: "+datasetInfo.name+"</small></p>");
 
         $( "#pane1" ).html(htmls.join(""));
 
@@ -891,9 +1007,9 @@ var cellbrowser = function() {
 
     }
 
-    function buildListPanel(datasetList, noteSpace, listGroupHeight, leftPaneWidth, htmls, selName) {
+    function buildListPanel(datasetList, listGroupHeight, leftPaneWidth, htmls, selName) {
         /* make a dataset list and append its html lines to htmls */
-        htmls.push("<div id='tpDatasetList' class='list-group' style='width:400px; position:absolute; top:"+noteSpace+"; height:"+listGroupHeight+"px; overflow-y:scroll; width:"+leftPaneWidth+"px'>");
+        htmls.push("<div id='tpDatasetList' class='list-group' style='float: left; margin-top: 1em; height:"+listGroupHeight+"px; overflow-y:scroll; width:"+leftPaneWidth+"px'>");
         if (!datasetList || datasetList.length===0) {
             alert("No datasets are available. Please make sure that at least one dataset does not set visibility=hide "+
                 " or that at least one collection is defined. Problems? -> cells@ucsc.edu");
@@ -913,11 +1029,38 @@ var cellbrowser = function() {
             }
 
             var bodyPartStr = "";
+            var disStr = "";
+            var orgStr = "";
+            var projStr = "";
+            var domStr = "";
+            var lifeStr = "";
+
             if (dataset.body_parts) {
                 bodyPartStr = dataset.body_parts.join("|");
             }
+            if (dataset.diseases) {
+                disStr = dataset.diseases.join("|");
+            }
+            if (dataset.organisms) {
+                orgStr = dataset.organisms.join("|");
+            }
+            if (dataset.projects) {
+                projStr = dataset.projects.join("|");
+            }
+            if (dataset.domains) {
+                domStr = dataset.domains.join("|");
+            }
+            if (dataset.life_stages) {
+                lifeStr = dataset.lifeStages.join("|");
+            }
 
-            var line = "<a id='tpDatasetButton_"+i+"' data-body-parts='"+bodyPartStr+"' role='button' class='tpListItem list-group-item "+clickClass+"' data-datasetid='"+i+"'>"; // bootstrap seems to remove the id
+            var line = "<a id='tpDatasetButton_"+i+"' "+"data-body='"+bodyPartStr+"' "+
+                "data-dis='"+disStr+"' "+
+                "data-org='"+orgStr+"' "+
+                "data-proj='"+projStr+"' "+
+                "data-dom='"+domStr+"' "+
+                "data-stage='"+lifeStr+"' "+
+                "role='button' class='tpListItem list-group-item "+clickClass+"' data-datasetid='"+i+"'>"; // bootstrap seems to remove the id
             htmls.push(line);
 
             if (!dataset.isSummary)
@@ -938,67 +1081,73 @@ var cellbrowser = function() {
 
             if (dataset.tags!==undefined) {
                 for (var tagI = 0; tagI < dataset.tags.length; tagI++) {
-                var tag = dataset.tags[tagI];
-                htmls.push("<span class='badge'>"+tag+"</span>");
+                    var tag = dataset.tags[tagI];
+                    if (tag==="smartseq2" || tag==="ATAC" || tag==="10x")
+                        continue
+                    htmls.push("<span class='badge'>"+tag+"</span>");
                 }
             }
             htmls.push(dataset.shortLabel+"</a>");
-            //if (db!==null && db.name===dataset.name)
-                //activeIdx = i;
         }
         htmls.push("</div>"); // list-group
         return selIdx;
     }
 
-    function getBodyParts(datasets) {
+    function getDatasetAttrs(datasets, attrName) {
         /* return list of body_parts given a dataset array */
         var bpObj = {};
         for (let i=0; i < datasets.length; i++) {
             let ds = datasets[i];
-            if (ds.body_parts===undefined)
+            if (ds[attrName]===undefined)
                 continue
-            for (let bp of ds.body_parts)
+            for (let bp of ds[attrName])
                 bpObj[bp] = true;
         }
 
-        let bodyParts = keys(bpObj);
-        bodyParts.sort();
-        return bodyParts;
+        let allValues = keys(bpObj);
+        allValues.sort();
+        return allValues;
     }
 
-    function filterDatasetsDom(onlyBps) {
-        /* keep only datasets with a a body_tag in filtNames */
+    function filterDatasetsDom() {
+        /* keep only datasets that fulfill the filters */
+
+        // read the current filter values of the dropboxes
+        var categories = ["Body", "Dis", "Org", "Proj"];
+        var filtVals = {};
+        for (var category of categories)
+            filtVals[category] = $("#tp"+category+"Combo").val();
 
         let elList = $(".tpListItem");
         for (let el of elList) {
-            let bpStr = el.getAttribute("data-body-parts");
-            let found = false;
-            if (!onlyBps || onlyBps.length==0)
-                // if no filtering is specified just show everything
-                found = true;
-            else
-                if (bpStr && bpStr!=="") {
-                    let bps = bpStr.split("|");
-                    if (onlyBps.length===0)
+            // never touch the first/summary element
+            if (el.getAttribute("data-body")==="summary")
+                continue
+            // read the values of the current DOM element
+            var domVals = {};
+            for (var category of categories)
+                domVals[category] = el.getAttribute("data-"+category.toLowerCase());
+
+            var isShown = true;
+            // now compare filtVals and domVals
+            for (var category of categories) {
+                var filtList = filtVals[category];
+                var domList = domVals[category];
+
+                let found = false;
+                if (filtList.length===0)
+                    found = true;
+                for (var filtVal of filtList)
+                    if (domList.indexOf(filtVal)!=-1)
                         found = true;
-                    else {
-                        if (bps.indexOf("summary")!==-1) { // never filter the summary
-                            found = true;
-                        }
-                        //if (bps.indexOf("all")!==-1) { // always match datasets with "all"
-                            //found = true;
-                        //}
-                        else
-                            for (let bp of onlyBps) {
-                                if (bps.indexOf(bp)!==-1) {
-                                    found = true;
-                                    break;
-                                }
-                            }
-                    }
-                }
 
-            if (found)
+                if (!found) {
+                    isShown = false;
+                    break;
+                }
+            }
+            
+            if (isShown)
                 el.style.display="";
             else
                 el.style.display="none";
@@ -1015,7 +1164,6 @@ var cellbrowser = function() {
         var listGroupHeight = 0;
         var leftPaneWidth = 400;
         var title = "Choose Cell Browser Dataset";
-        var noteSpace = "3em"; // space from top of dialog to info pane and tabs
 
         // inline functions
         function openCollOrDataset(selDatasetIdx) {
@@ -1030,6 +1178,19 @@ var cellbrowser = function() {
             //changeUrl({"bp":null});
         }
 
+        function buildFilter(html, filterVals, filterLabel, urlVar, comboId, comboLabel) {
+            /* build html for a faceting filter */
+            if (filterVals.length==0)
+                return false;
+            html.push("<span style='margin-right:5px'>"+filterLabel+":</span>");
+            let selPar = getVarSafe(urlVar);
+            if (selPar && selPar!=="")
+                filtList = selPar.split("|");
+            buildComboBox(html, comboId, filterVals, filtList, comboLabel, 200, {multi:true});
+            html.push("&nbsp;&nbsp;");
+            return true;
+        }
+
         function connectOpenPane(selDatasetIdx, datasetList) {
             /* set all the click handlers for the left open dataset pane */
             $("button.list-group-item").eq(selDatasetIdx).css("z-index", "1000"); // fix up first overlap
@@ -1066,13 +1227,26 @@ var cellbrowser = function() {
             });
         }
 
-        function onBodyPartChange(ev) {
-            /* called when user changes body part list */
-            let filtNames = $("#tpBodyCombo").val();
+        function onFilterChange(ev) {
+            /* called when user changes a filter: updates list of datasets shown */
+            var filtNames = $(this).val();
+
+            var param = null;
+            if (this.id==="tpBodyCombo")
+                param = "bp";
+            else if (this.id=="tpDisCombo")
+                param = "dis";
+            else if (this.id=="tpOrgCombo")
+                param = "org";
+            else if (this.id=="tpProjCombo")
+                param = "proj";
+
             // change the URL
-            let filtArg = filtNames.join("_");
-            changeUrl({"bp":filtArg});
-            filterDatasetsDom(filtNames);
+            var filtArg = filtNames.join("~");
+            var urlArgs = {}
+            urlArgs[param] = filtArg;
+            changeUrl(urlArgs);
+            filterDatasetsDom();
         }
 
         // -- end inline functions
@@ -1100,20 +1274,35 @@ var cellbrowser = function() {
             changeUrl({"ds":openDsInfo.name.replace(/\//g, " ")}); // + is easier to type
         }
 
-        let doFaceting = false;
+        let doFilters = false;
         let filtList = [];
+        let bodyParts = null;
+        let diseases = null;
+        let organisms = null;
+        let projects = null;
+        let domains = null
+        let lifeStages = null;
         if (openDsInfo.parents === undefined && openDsInfo.datasets !== undefined) {
-            //noteLines.push("<span>Filter:</span>");
-            let bodyParts = getBodyParts(openDsInfo.datasets);
-            if (bodyParts.length!==0) {
-                noteLines.push("<span style='margin-right:5px'>Filter datasets by organ:</span>");
-                doFaceting = true;
-                // some mirrors don't use the "body_parts" statement and don't need the faceting
-                let selPar = getVarSafe("bp");
-                if (selPar && selPar!=="")
-                    filtList = selPar.split("_");
-
-                buildComboBox(noteLines, "tpBodyCombo", bodyParts, filtList, "select organs...", 200, {multi:true});
+            bodyParts = getDatasetAttrs(openDsInfo.datasets, "body_parts");
+            diseases = getDatasetAttrs(openDsInfo.datasets, "diseases");
+            organisms = getDatasetAttrs(openDsInfo.datasets, "organisms");
+            projects = getDatasetAttrs(openDsInfo.datasets, "projects");
+            domains = getDatasetAttrs(openDsInfo.datasets, "domains");
+            lifeStages = getDatasetAttrs(openDsInfo.datasets, "life_stages");
+
+            // mirror websites are not using the filters at all. So switch off the entire UI for it then.
+            if (bodyParts.length!==0 || disease.length!==0 || organisms.length!==0 || projects.length!==0 || domains.length!==0 || lifeStages.length!==0)
+                doFilters = true;
+
+            if (doFilters) {
+                noteLines.push("<div style='margin-right: 10px; font-weight: bold'>Filters:</div>");
+
+                buildFilter(noteLines, bodyParts, "Organ", "body", "tpBodyCombo", "select organs...");
+                buildFilter(noteLines, diseases, "Disease", "dis", "tpDisCombo", "select diseases...");
+                buildFilter(noteLines, organisms, "Species", "org", "tpOrgCombo", "select species...");
+                buildFilter(noteLines, projects, "Project", "proj", "tpProjCombo", "select project...");
+                buildFilter(noteLines, domains, "Scient. Domain", "dom", "tpProjCombo", "select domain...");
+                buildFilter(noteLines, lifeStages, "Life Stages", "stage", "tpProjCombo", "select stage...");
             }
         }
 
@@ -1136,7 +1325,6 @@ var cellbrowser = function() {
                 backLinks.push("<span class='tpBackLink link' data-open-dataset='"+parName+"' data-sel-dataset='"+childName+"'>"+parLabel+"</span>");
             }
             noteLines.push(backLinks.join("&nbsp;&gt;&nbsp;"));
-            noteSpace = "5em"; // TODO: redesign dialog to not have hard-coded spacing
         }
 
         if (onlyInfo)
@@ -1154,19 +1342,23 @@ var cellbrowser = function() {
 
         var winWidth = window.innerWidth - 0.05*window.innerWidth;
         var winHeight = window.innerHeight - 0.05*window.innerHeight;
-        var tabsWidth = winWidth - leftPaneWidth - 70;
+        var tabsWidth = winWidth - leftPaneWidth - 50;
         listGroupHeight = winHeight - 100;
 
         var htmls = ["<div style='line-height: 1.1em'>"];
         htmls.push(noteLines.join(""));
         htmls.push("</div>");
 
-        if (onlyInfo)
+        htmls.push("<div id='tpDatasetBrowser'>");
+
+        if (onlyInfo) {
             leftPaneWidth = 0;
-        else
-            activeIdx = buildListPanel(datasetList, noteSpace, listGroupHeight, leftPaneWidth, htmls, selName);
+            htmls.push("<div></div>"); // keep structure of the page the same, skip the left pane
+        } else {
+            activeIdx = buildListPanel(datasetList, listGroupHeight, leftPaneWidth, htmls, selName);
+            htmls.push("<div id='tpOpenDialogDatasetDesc' style='width:"+tabsWidth+"px; float:right; left: " + (leftPaneWidth + 10) + "px; margin-top: 1em; border: 0'>");
+        }
 
-        htmls.push("<div id='tpOpenDialogDatasetDesc' style='width:"+tabsWidth+"px; position:absolute; left: " + (leftPaneWidth + 20) + "px; top: "+noteSpace+"; border: 0'>");
         htmls.push("<div id='tpOpenDialogTabs' style='border: 0'>");
         htmls.push("<ul class='nav nav-tabs'>");
         htmls.push("<li class='active'><a class='tpDatasetTab' id='tabLink1' data-toggle='tab' href='#pane1'>Abstract</a></li>");
@@ -1196,6 +1388,8 @@ var cellbrowser = function() {
         htmls.push("</div>"); // tpOpenDialogDatasetDesc
 
         //htmls.push("<div id='tpSelectedId' data-selectedid='0'>"); // store the currently selected datasetId in the DOM
+        htmls.push("</div>"); // tpDatasetBrowser
+
         var selDatasetIdx = 0;
 
         var buttons = [];
@@ -1219,9 +1413,23 @@ var cellbrowser = function() {
 
         $("#tpOpenDialogTabs").tabs();
 
-        if (doFaceting) {
-            activateCombobox("tpBodyCombo", 200);
-            $("#tpBodyCombo").change( onBodyPartChange );
+        if (doFilters) {
+            if (bodyParts) {
+                activateCombobox("tpBodyCombo", 200);
+                $("#tpBodyCombo").change( onFilterChange );
+            }
+            if (diseases) {
+                activateCombobox("tpDisCombo", 200);
+                $("#tpDisCombo").change( onFilterChange );
+            }
+            if (organisms) {
+                activateCombobox("tpOrgCombo", 200);
+                $("#tpOrgCombo").change( onFilterChange );
+            }
+            if (projects) {
+                activateCombobox("tpProjCombo", 200);
+                $("#tpProjCombo").change( onFilterChange );
+            }
         }
 
         $('.tpBackLink').click( function(ev) {
@@ -1247,7 +1455,7 @@ var cellbrowser = function() {
         }
 
         if (getVarSafe("ds")===undefined) // only filter on the top level
-            filterDatasetsDom(filtList);
+            filterDatasetsDom();
         connectOpenPane(selDatasetIdx, datasetList);
         // finally, activate the default pane and load its html
         openDatasetLoadPane(openDsInfo);
@@ -1355,15 +1563,6 @@ var cellbrowser = function() {
         htmls.push('</input>');
 
         htmls.push('<select style="max-width: 300px" id="tpSelectMetaValueEnum_'+rowIdx+'" name="metaValue">');
-        // if ("m" in queryExpr) {
-          //   var op = getQueryOp(queryExpr);
-            // var fieldName = query["m"];
-            //var fieldValue = queryExpr[op];
-            //var metaInfo = findMetaInfo(fieldName);
-            //for (var valIdx = 0; valIdx < metaInfo.valCounts.length; valIdx++) {
-            //    htmls.push('<option value="'+valIdx+'">'+metaInfo.valCounts[i][0]+'</option>');
-           // }
-       // }
         htmls.push('</select>');
         htmls.push('</div>'); // tpSelectRow_<rowIdx>
 
@@ -2303,10 +2502,11 @@ var cellbrowser = function() {
 
         htmls.push("<p><b>Version:</b> "+gVersion+"</p>");
         htmls.push("<p><b>Written by:</b> Maximilian Haeussler, Nikolay Markov (U Northwestern), Brian Raney, Lucas Seninge</p>");
-        htmls.push("<p><b>Testing / User interface / Documentation / Data import / User support:</b> Matt Speir</p>");
+        htmls.push("<p><b>Testing / User interface / Documentation / Data import / User support:</b> Matt Speir, Brittney Wick</p>");
         htmls.push("<p><b>Code contributions by:</b> Pablo Moreno (EBI, UK)</p>");
-        htmls.push("<p><b>Documentation:</b> <a target=_blank href='https://cellbrowser.readthedocs.io/'>Readthedocs</a></p>");
-        htmls.push("<p><b>Github Repo: </b><a target=_blank href='https://github.com/maximilianh/cellBrowser/'>cellBrowser</a></p>");
+        htmls.push("<p><b>Documentation:</b> <a class='link' target=_blank href='https://cellbrowser.readthedocs.io/'>Readthedocs</a></p>");
+        htmls.push("<p><b>Github Repo: </b><a class='link' target=_blank href='https://github.com/maximilianh/cellBrowser/'>cellBrowser</a></p>");
+        htmls.push("<p><b>Paper: </b><a class='link' target=_blank href='https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btab503/6318386'>Speir et al, Bioinformatics 2021, DOI:10.1093/bioinformatics/btab503/6318386</a></p>");
 
         showDialogBox(htmls, title, {showClose:true, height:dlgHeight, width:500});
     }
@@ -2415,7 +2615,8 @@ var cellbrowser = function() {
          htmls.push('<a href="#" class="dropdown-toggle" data-toggle="dropdown" data-submenu role="button" aria-haspopup="true" aria-expanded="false">Help</a>');
          htmls.push('<ul class="dropdown-menu">');
          htmls.push('<li><a href="#" id="tpAboutButton">About</a></li>');
-         htmls.push('<li><a href="#" id="tpTutorialButton">Tutorial</a></li>');
+         htmls.push('<li><a href="https://cellbrowser.readthedocs.io/en/master/interface.html" target=_blank id="tpQuickstartButton">How to use this website</a></li>');
+         htmls.push('<li><a href="#" id="tpTutorialButton">Interactive Tutorial</a></li>');
          htmls.push('<li><a target=_blank href="https://github.com/maximilianh/cellBrowser#readme" id="tpGithubButton">Setup your own cell browser</a></li>');
          htmls.push('</ul>'); // Help dropdown-menu
          htmls.push('</li>'); // Help dropdown container
@@ -2573,7 +2774,7 @@ var cellbrowser = function() {
 
 
     function colorByMetaField(fieldName, doneLoad) {
-        /* load the meta data for a field, setup the colors, send it all to the renderer and call doneLoad */
+        /* load the meta data for a field, setup the colors, send it all to the renderer and call doneLoad. if doneLoad is undefined, redraw everything  */
 
         function onMetaArrLoaded(metaArr, metaInfo) {
             gLegend = buildLegendForMeta(metaInfo);
@@ -2645,7 +2846,6 @@ var cellbrowser = function() {
             db.loadMetaVec(metaInfo, onMetaArrLoaded, onProgress);
 
 
-       //changeUrl({"gene":null, "meta":fieldName});
        changeUrl({"pal":null});
        // clear the gene search box
        var select = $('#tpGeneCombo')[0].selectize.clear();
@@ -2918,7 +3118,6 @@ var cellbrowser = function() {
                 changeUrl({"locus":locusStr, "meta":null});
             } else {
                 changeUrl({"gene":locusStr, "meta":null});
-                //selectizeSetValue("#tpGeneCombo", locusStr);
             }
 
             makeLegendExpr(fullLocusStr, geneDesc, binInfo, exprArr, decArr);
@@ -2935,6 +3134,7 @@ var cellbrowser = function() {
                     break;
                 }
             }
+
             gRecentGenes.unshift([locusStr, geneDesc]); // insert at position 0
             gRecentGenes = gRecentGenes.slice(0, 9); // keep only nine last
             buildGeneTable(null, "tpRecentGenes", null, null, gRecentGenes);
@@ -2949,41 +3149,93 @@ var cellbrowser = function() {
 
     }
 
-   function gotCoords(coords, info, clusterInfo, newRadius) {
-       /* called when the coordinates have been loaded */
-       if (coords.length===0)
-           alert("cellBrowser.js/gotCoords: coords.bin seems to be empty");
-       var opts = {};
-       if (newRadius)
-           opts["radius"] = newRadius;
+    function gotCoords(coords, info, clusterInfo, newRadius) {
+        /* called when the coordinates have been loaded */
+        if (coords.length===0)
+            alert("cellBrowser.js/gotCoords: coords.bin seems to be empty");
+        var opts = {};
+        if (newRadius)
+            opts["radius"] = newRadius;
 
-       // label text can be overriden by the user cart
-       var labelField = db.conf.labelField;
+        // label text can be overriden by the user cart
+        var labelField = db.conf.labelField;
 
-       if (clusterInfo) {
-           var origLabels = [];
-           var clusterMids = clusterInfo.labels;
-           // old-style files contain just coordinates, no order
-           if (clusterMids===undefined)
-               clusterMids = clusterInfo;
+        if (! db.gLabelCoordCache )
+            db.gLabelCoordCache = {};
 
-           for (var i = 0; i < clusterMids.length; i++) {
-               var labelInfo = clusterMids[i];
-               var oldName = labelInfo[2];
-               origLabels.push(oldName);
-               var newName = oldName;
-               labelInfo[2] = newName;
-           }
-           renderer.origLabels = origLabels;
-           db.clusterOrder = clusterInfo.order;
+        if (clusterInfo) {
+            var origLabels = [];
+            var clusterMids = clusterInfo.labels;
+            // old-style files contain just coordinates, no order
+            if (clusterMids === undefined) {
+                clusterMids = clusterInfo;
+            }
 
+            db.gLabelCoordCache[labelField] = clusterMids;
+            for (var i = 0; i < clusterMids.length; i++) {
+                origLabels.push(clusterMids[i][2]);
+            }
+            renderer.origLabels = origLabels;
+            clusterMids = [];
+         }
+
+        opts["lines"] = clusterInfo.lines;
+        opts["lineWidth"] = db.conf.lineWidth;
+
+        renderer.setCoords(coords, clusterMids, info.minX, info.maxX, info.minY, info.maxY, opts);
+    }
+
+    function computeAndSetLabels(values, metaInfo) {
+        /* recompute the label positions and redraw everything. Uses a cache for the label positions */
+        var labelCoords;
+        if (db.gLabelCoordCache[metaInfo.label] !== undefined) {
+            labelCoords = db.gLabelCoordCache[metaInfo.label];
+        } else {
+            var coords = renderer.coords.orig;
+            var names = null;
+            if (metaInfo.type !== "float" && metaInfo.type !== "int") {
+                var names = metaInfo.ui.shortLabels;
+            }
+
+            // console.log(metaInfo);
+
+            console.time("cluster centers");
+            var calc = {};
+            for (var i = 0, I = values.length; i < I; i++) {
+                if (names) {
+                    var label = names[values[i]];
+                } else {
+                    var label = metaInfo.origVals[i].toFixed(2);
+                }
+                if (calc[label] === undefined) {
+                    calc[label] = [[], [], 0]; // all X, all Y, count
+                }
+                calc[label][0].push(coords[i * 2]);
+                calc[label][1].push(coords[i * 2 + 1]);
+                calc[label][2] += 1;
+            }
+            labelCoords = [];
+            for (label in calc) {
+                var midX = selectMedian(calc[label][0]);
+                var midY = selectMedian(calc[label][1]);
+                labelCoords.push([midX, midY, label]);
+            }
+            console.timeEnd("cluster centers");
+            db.gLabelCoordCache[metaInfo.label] = labelCoords;
         }
+        renderer.setLabelCoords(labelCoords);
+    }
 
-       opts["lines"] = clusterInfo.lines;
-       opts["lineWidth"] = db.conf.lineWidth;
+    function setLabelField(labelField) {
+        /* change the field that is used for drawing the labels. Do not redraw. */
+        var metaInfo = db.findMetaInfo(labelField);
+        db.conf.activeLabelField = labelField;
 
-       renderer.setCoords(coords, clusterMids, info.minX, info.maxX, info.minY, info.maxY, opts);
-   }
+        if (metaInfo.arr) // preloaded
+            computeAndSetLabels(metaInfo.arr, metaInfo);
+        else
+            db.loadMetaVec(metaInfo, computeAndSetLabels);
+    }
 
    function colorByDefaultField(onDone) {
        /* get the default coloring field from the config or the URL and start coloring by it.
@@ -3058,6 +3310,7 @@ var cellbrowser = function() {
            loadsDone +=1;
            if (loadsDone===2) {
                buildLegendBar();
+               setLabelField(db.conf.labelField);
 
                if (forcePalName!==null) {
                    legendChangePaletteAndRebuild(forcePalName);
@@ -3322,7 +3575,9 @@ var cellbrowser = function() {
             htmls.push('<ul class="dropdown-menu pull-right">');
             htmls.push('<li><a class="tpColorLink" data-palette="default" href="#">Reset to Default</a></li>');
             htmls.push('<li><a class="tpColorLink" data-palette="rainbow" href="#">Qualitative: Rainbow</a></li>');
-            htmls.push('<li><a class="tpColorLink" data-palette="tol-dv" href="#">Qualitative: Paul Tol&#39;s</a></li>');
+            htmls.push('<li><a class="tpColorLink" data-palette="tatarize" href="#">Qualitative: Tatarize</a></li>');
+            htmls.push('<li><a class="tpColorLink" data-palette="iwanthue" href="#">Qualitative: Iwanthue</a></li>');
+            htmls.push('<li><a class="tpColorLink" data-palette="tol-dv" href="#">Semi-Qualitative: Paul Tol&#39;s</a></li>');
             //htmls.push('<li><a class="tpColorLink" data-palette="cb-Paired" href="#">Qualitative: paired</a></li>');
             //htmls.push('<li><a class="tpColorLink" data-palette="cb-Set3" href="#">Qualitative: pastel</a></li>');
             htmls.push('<li><a class="tpColorLink" data-palette="blues" href="#">Gradient: shades of blue</a></li>');
@@ -3660,6 +3915,7 @@ var cellbrowser = function() {
         var saneSym = event.target.id.split("_")[1]; // the symbol of the gene, as only-alphaNum chars
         $('.tpMetaBox').removeClass('tpMetaSelect');
         $('.tpGeneBarCell').removeClass("tpGeneBarCellSelected");
+        // XX TODO: How find all the elements with this ID?
         $('#tpGeneBarCell_'+saneSym).addClass("tpGeneBarCellSelected");
 
         // search through both quick and recent gene lists to find the real gene symbol
@@ -3670,6 +3926,15 @@ var cellbrowser = function() {
         if (geneSym===null)
             geneSym = alphaNumSearch(gRecentGenes, saneSym);
 
+        // if the gene symbol was actually a gene identifier, than no search will have succeeded
+        // This is the more "modern" way of doing things: the ID will have a gene identifier
+        // and the tag will have been the symbol. This is because most modern matrices have both
+        // gene ID and symbol and the index will also have both and so the user can search for both.
+        // (in the old days, see code above, there were only symbols)
+        // So in this case, fall back to the identifier
+        if (geneSym===null)
+            geneSym = saneSym;
+
         colorByLocus(geneSym);
         event.stopPropagation();
     }
@@ -3927,7 +4192,6 @@ var cellbrowser = function() {
         }
 
         var tableWidth = metaBarWidth;
-        //var cellWidth = gGeneCellWidth;
 
         if (title) {
             htmls.push("<div style='margin-top:8px' id='"+divId+"_title'>");
@@ -3953,34 +4217,32 @@ var cellbrowser = function() {
             return;
         }
 
-        htmls.push('<table style="margin-top:10px" id="tpGeneTable"><tr>');
-        //htmls.push('<td><button id="tpChangeGenes" title="Change the list of genes that are displayed in this table" class = "ui-button ui-widget ui-corner-all" style="width:95%">Change</button></td>');
-
         // need max length of gene names to make number of columns
-        var maxLen = 0;
-        for (var i=0; i < geneInfos.length; i++) {
-            var geneId = geneInfos[i][0];
-            maxLen = Math.max(maxLen, geneId.length);
-        }
-
-        var colsPerRow = Math.floor(40.0/maxLen);
-        //var colsPerRow = Math.round(tableWidth / cellWidth);
-        var cellWidth = Math.round(tableWidth/colsPerRow);
-
-        var currWidth = 1;
+        //var currWidth = 1;
         var i = 0;
         while (i < geneInfos.length) {
             var geneInfo = geneInfos[i];
-            var geneId   = geneInfo[0];
+            var geneIdOrSym   = geneInfo[0];
             var geneDesc = geneInfo[1];
-            if (((i % colsPerRow) === 0) && (i!==0)) {
-                htmls.push("</tr><tr>");
+            if (geneDesc===undefined)
+                geneDesc = geneId;
+
+            // geneIdOrSym can be just the symbol (if we all we have is symbols) or geneId|symbol
+            var geneId
+            var sym;
+            if (geneIdOrSym.indexOf("|")!==-1) {
+                var parts = geneIdOrSym.split("|");
+                geneId = parts[0];
+                sym = parts[1];
+            } else {
+                geneId = geneIdOrSym;
+                sym = geneId;
             }
-            if (geneId in db.geneOffsets)
-                htmls.push('<td title="'+geneDesc+'" id="tpGeneBarCell_'+onlyAlphaNum(geneId)+'" class="tpGeneBarCell">'+geneId+'</td>');
+
+            //if (geneId in db.geneOffsets)
+            htmls.push('<span title="'+geneDesc+'" style="width: fit-content;" id="tpGeneBarCell_'+onlyAlphaNum(geneId)+'" class="tpGeneBarCell">'+sym+'</span>');
             i++;
         }
-        htmls.push("</tr></table>");
         htmls.push("</div>"); // divId
 
         if (doUpdate) {
@@ -3988,6 +4250,18 @@ var cellbrowser = function() {
         }
     }
 
+    function resizeGeneTableDivs() {
+        /* the size of the DIVs in the gene table depends on the size of the longest DIV in pixels and we know that only once the table is shown. so resize here now */
+        var tdEls = document.getElementById("tpGenes").querySelectorAll("span");
+        var maxWidth = 0;
+        for (var el of tdEls) {
+            maxWidth = Math.max(maxWidth, el.offsetWidth);
+        }
+        for (var el of tdEls) {
+            el.style.minWidth = maxWidth+"px";
+        }
+    }
+
     function likeEmptyString(label) {
     /* some special values like "undefined" and empty string get colored in grey  */
         return (label===null || label.trim()==="" || label==="none" || label==="None" || label==="unknown" ||
@@ -4377,7 +4651,7 @@ var cellbrowser = function() {
         htmls.push('</select>');
     }
 
-    function loadCoordSet(coordIdx) {
+    function loadCoordSet(coordIdx, labelFieldName) {
         /* load coordinates and color by meta data */
         var newRadius = db.conf.coords[coordIdx].radius;
         var colorOnMetaField = db.conf.coords[coordIdx].colorOnMeta;
@@ -4385,8 +4659,11 @@ var cellbrowser = function() {
         db.loadCoords(coordIdx,
                 function(coords, info, clusterMids) {
                     gotCoords(coords,info,clusterMids, newRadius);
+
+                    setLabelField(labelFieldName);
+
                     if (colorOnMetaField!==undefined)
-                        colorByMetaField(colorOnMetaField);
+                        colorByMetaField(colorOnMetaField, undefined);
                     else
                         renderer.drawDots();
                 },
@@ -4396,9 +4673,15 @@ var cellbrowser = function() {
     function onLayoutChange(ev, params) {
         /* user changed the layout in the combobox */
         var coordIdx = parseInt(params.selected);
-        loadCoordSet(coordIdx);
+
+        var labelFieldIdx = parseInt($("#tpLabelCombo").val().split("_")[1]);
+        var labelFieldName = db.getMetaFields()[labelFieldIdx].name;
+
+        loadCoordSet(coordIdx, labelFieldName);
+
         changeUrl({"layout":coordIdx, "zoom":null});
         renderer.coordIdx = coordIdx;
+
         // remove the focus from the combo box
         removeFocus();
     }
@@ -4433,6 +4716,14 @@ var cellbrowser = function() {
         colorByMetaField(fieldName);
     }
 
+    function onLabelComboChange(ev, choice) {
+        /* called when user changes the label field combo box */
+        var fieldId = parseInt(choice.selected.split("_")[1]);
+        var fieldName = db.getMetaFields()[fieldId].name;
+        setLabelField(fieldName);
+        renderer.drawDots();
+    }
+
     function showCollectionDialog(collName) {
         /* load collection with given name and open dialog box for it */
         loadCollectionInfo(collName, function(collData) { openDatasetDialog(collData)});
@@ -4572,20 +4863,27 @@ var cellbrowser = function() {
         htmls.push('</div>');
     }
 
-    function buildMetaFieldCombo(htmls, idOuter, id, left) {
+    function buildMetaFieldCombo(htmls, idOuter, id, left, selectedField) {
         var metaFieldInfo = db.getMetaFields();
         htmls.push('<div id="'+idOuter+'" style="padding-left:2px; display:inline">');
-        var entries = [["_none", ""]];
+        //var entries = [["_none", ""]];
+        var entries = [];
+        var selIdx = 0;
         for (var i = 1; i < metaFieldInfo.length; i++) { // starts at 1, skip ID field
             var field = metaFieldInfo[i];
             var fieldName = field.label;
-            //var hasTooManyVals = (field.diffValCount>100);
-            //if (hasTooManyVals)
-                //continue;
+            // cannot label on something that is a number or has a ton of values
+            var hasTooManyVals = (field.diffValCount>100 || field.type==="int" || field.type==="float");
+            if (hasTooManyVals)
+                continue;
+
             entries.push( ["tpMetaVal_"+i, fieldName] );
+            if (selectedField == fieldName) {
+                selIdx = i;
+            }
         }
 
-        buildComboBox(htmls, id, entries, 0, "select a field...", metaBarWidth+50);
+        buildComboBox(htmls, id, entries, selIdx, "select a field...", metaBarWidth+50);
         htmls.push('</div>');
     }
 
@@ -5282,8 +5580,12 @@ var cellbrowser = function() {
         htmls.push("</ul>");
 
         htmls.push("<div id='tpAnnotTab'>");
+
         htmls.push('<label style="padding-left: 2px; margin-bottom:8px; padding-top:8px" for="'+"tpMetaCombo"+'">Color by Annotation</label>');
         buildMetaFieldCombo(htmls, "tpMetaComboBox", "tpMetaCombo", 0);
+        htmls.push('<label style="padding-left: 2px; margin-bottom:8px; padding-top:8px" for="tpLabelCombo">Label by non-num. Annotation</label>');
+        buildMetaFieldCombo(htmls, "tpLabelComboBox", "tpLabelCombo", 0, db.conf.labelField);
+
         htmls.push('<div style="padding-top:4px; padding-bottom: 4px; padding-left:2px" id="tpHoverHint" class="tpHint">Hover over a '+gSampleDesc+' to update data below</div>');
         htmls.push('<div style="padding-top:4px; padding-bottom: 4px; padding-left:2px; display: none" id="tpSelectHint" class="tpHint">Cells selected. No update on hover.</div>');
 
@@ -5301,11 +5603,13 @@ var cellbrowser = function() {
             buildPeakList(htmls);
 
         var geneLabel = getGeneLabel();
-        buildGeneTable(htmls, "tpRecentGenes", "Recent "+geneLabel+"s", "Hover or select cells to update colors", gRecentGenes);
+        buildGeneTable(htmls, "tpRecentGenes", "Recent "+geneLabel+"s",
+            "Hover or select cells to update colors", gRecentGenes);
 
-        //var myGenes = loadMyGenes();
+        // var myGenes = loadMyGenes();
 
-        var noteStr = "No genes or peaks defined. Use the setting quickGenesFile in cellbrowser.conf to add a file with gene symbols or peaks that will be shown here";
+        var noteStr = "No genes or peaks defined. Use the setting quickGenesFile in "
+            "cellbrowser.conf to add a file with gene symbols or peaks that will be shown here";
         buildGeneTable(htmls, "tpGenes", "Dataset "+geneLabel+"s", null, db.conf.quickGenes, noteStr);
 
         htmls.push("</div>"); // tpGeneTab
@@ -5314,6 +5618,8 @@ var cellbrowser = function() {
 
         $(document.body).append(htmls.join(""));
 
+        resizeGeneTableDivs();
+
         $("#tpLeftTabs").tabs();
         $('#tpLeftTabs').tabs("option", "active", 0); // open the first tab
 
@@ -5322,6 +5628,9 @@ var cellbrowser = function() {
 
         activateCombobox("tpMetaCombo", metaBarWidth-10);
         $("#tpMetaCombo").change( onMetaComboChange );
+
+        activateCombobox("tpLabelCombo", metaBarWidth-10);
+        $("#tpLabelCombo").change( onLabelComboChange );
         connectMetaPanel();
     }
 
@@ -5478,6 +5787,24 @@ var cellbrowser = function() {
         return pal;
     }
 
+    function makeTatarizePalette(n) {
+    /* suggested by Niko Papadopoulos from Detlev Arendt's group see http://godsnotwheregodsnot.blogspot.com/2012/09/color-distribution-methodology.html */
+        var pal = ["000000", "FFFF00", "1CE6FF", "FF34FF", "FF4A46", "008941", "006FA6", "A30059",
+                "FFDBE5", "7A4900", "0000A6", "63FFAC", "B79762", "004D43", "8FB0FF", "997D87",
+                "5A0007", "809693", "FEFFE6", "1B4400", "4FC601", "3B5DFF", "4A3B53", "FF2F80",
+                "61615A", "BA0900", "6B7900", "00C2A0", "FFAA92", "FF90C9", "B903AA", "D16100",
+                "DDEFFF", "000035", "7B4F4B", "A1C299", "300018", "0AA6D8", "013349", "00846F",
+                "372101", "FFB500", "C2FFED", "A079BF", "CC0744", "C0B9B2", "C2FF99", "001E09",
+                "00489C", "6F0062", "0CBD66", "EEC3FF", "456D75", "B77B68", "7A87A1", "788D66",
+                "885578", "FAD09F", "FF8A9A", "D157A0", "BEC459", "456648", "0086ED", "886F4C",
+                "34362D", "B4A8BD", "00A6AA", "452C2C", "636375", "A3C8C9", "FF913F", "938A81",
+                "575329", "00FECF", "B05B6F", "8CD0FF", "3B9700", "04F757", "C8A1A1", "1E6E00",
+                "7900D7", "A77500", "6367A9", "A05837", "6B002C", "772600", "D790FF", "9B9700",
+                "549E79", "FFF69F", "201625", "72418F", "BC23FF", "99ADC0", "3A2465", "922329",
+                "5B4534", "FDE8DC", "404E55", "0089A3", "CB7E98", "A4E804", "324E72", "6A3A4C"];
+        return pal.slice(0, n);
+    }
+
     function makeColorPalette(palName, n) {
     /* return an array with n color hex strings */
     /* Use Google's palette functions for now, first Paul Tol's colors, if that fails, use the usual HSV rainbow
@@ -5488,8 +5815,12 @@ var cellbrowser = function() {
             pal = makeHslPalette(0.6, n);
         else if (palName==="magma" || palName==="viridis" || palName==="inferno" || palName=="plasma")
             pal = makePercPalette(palName, n);
+        else if (palName==="iwanthue")
+            pal = iWantHue(n);
         else if (palName==="reds")
             pal = makeHslPalette(0.0, n);
+        else if (palName==="tatarize")
+            pal = makeTatarizePalette(n);
         else {
             if (n===2)
                 pal = ["ADD8E6","FF0000"];
@@ -5739,7 +6070,7 @@ var cellbrowser = function() {
     }
 
     function getClusterFieldInfo() {
-        var clusterField = db.conf.labelField;
+        var clusterField = db.conf.activeLabelField;
         var clusterMetaInfo = db.findMetaInfo(clusterField);
         return clusterMetaInfo;
     }
@@ -6130,7 +6461,6 @@ var cellbrowser = function() {
 	    if (isDark(color))
 		fontColor = "white";
             $("#tpGeneBarCell_"+onlyAlphaNum(sym)).css({"background-color": "#"+color, "color" : fontColor});
-	    //$("#tpGeneBarCell_"+i).css("color", fontColor);
         }
         console.timeEnd("avgCalc");
     }
@@ -6395,13 +6725,13 @@ var cellbrowser = function() {
     }
 
     function onClusterNameHover(clusterName, nameIdx, ev) {
-       /* user hovers over cluster label */
-       var labelLines = [clusterName];
+        /* user hovers over cluster label */
+        var labelLines = [clusterName];
 
-       var labelField = db.conf.labelField;
-       var metaInfo = db.findMetaInfo(labelField);
-       var longLabels = metaInfo.ui.longLabels;
-       if (longLabels) {
+        var labelField = db.conf.labelField;
+        var metaInfo = db.findMetaInfo(labelField);
+        var longLabels = metaInfo.ui.longLabels;
+        if (longLabels) {
             for (let i=0; i<longLabels.length; i++) {
                 let shortLabel = metaInfo.ui.shortLabels[i];
                 let longLabel = longLabels[i];
@@ -6410,20 +6740,22 @@ var cellbrowser = function() {
                     break;
                 }
             }
-       }
-
-       if (db.conf.topMarkers!==undefined) {
-            labelLines.push("Top enriched/depleted markers: "+db.conf.topMarkers[clusterName].join(", "));
         }
-       labelLines.push("");
 
-       if (db.conf.markers!==undefined)
-            labelLines.push("Click to show full marker gene list.");
+        if (labelField == db.conf.activeLabelField) {
+            if (db.conf.topMarkers!==undefined) {
+                labelLines.push("Top enriched/depleted markers: "+db.conf.topMarkers[clusterName].join(", "));
+            }
+            labelLines.push("");
 
-       if (db.conf.clusterPngDir!==undefined) {
-            var fullPath = cbUtil.joinPaths([db.name, db.conf.clusterPngDir, clusterName+".png"]);
-            labelLines.push("<img src='"+fullPath+"'>");
-       }
+            if (db.conf.markers!==undefined)
+                labelLines.push("Click to show full marker gene list.");
+
+            if (db.conf.clusterPngDir!==undefined) {
+                var fullPath = cbUtil.joinPaths([db.name, db.conf.clusterPngDir, clusterName+".png"]);
+                labelLines.push("<img src='"+fullPath+"'>");
+            }
+        }
 
         labelLines.push("Alt/Option-Click to select cluster; Shift-Click to add cluster to selection");
         showTooltip(ev.clientX+15, ev.clientY, labelLines.join("<br>"));
@@ -6650,12 +6982,20 @@ var cellbrowser = function() {
     function onClusterNameClick(clusterName, _, event) {
         /* build and open the dialog with the marker genes table for a given cluster */
         var metaInfo = getClusterFieldInfo();
-        var nameIdx = metaInfo.ui.shortLabels.indexOf(clusterName);
+        var isNumber = false;
+        var nameIdx = null;
+        if (metaInfo.type == "int" || metaInfo.type == "float") {
+            isNumber = true;
+        } else {
+            nameIdx = metaInfo.ui.shortLabels.indexOf(clusterName);
+        }
         if (event.altKey || event.shiftKey) {
             db.loadMetaVec(metaInfo, function(values) {
                 var clusterCells = [];
                 for (var i = 0, I = values.length; i < I; i++) {
-                    if (values[i] == nameIdx) {
+                    if (isNumber && metaInfo.origVals[i].toFixed(2) == clusterName) {
+                        clusterCells.push(i);
+                    } else if (!isNumber && values[i] == nameIdx) {
                         clusterCells.push(i);
                     }
                 }
@@ -6670,6 +7010,9 @@ var cellbrowser = function() {
             });
             return;
         }
+        if (db.conf.labelField != db.conf.activeLabelField) {
+            return;
+        }
 
         var tabInfo = db.conf.markers; // list with (label, subdirectory)
 
@@ -6856,8 +7199,13 @@ var cellbrowser = function() {
 
             htmls.push("<tr>");
             var geneId = row[0];
+
+            // old marker files still have the format geneId|sym, so tolerate this here
+            if (geneId.indexOf("|") > -1)
+                geneId = geneId.split("|")[0];
+
             var geneSym = row[1];
-            htmls.push("<td><a data-gene='"+geneSym+"' class='link tpLoadGeneLink'>"+geneSym+"</a>");
+            htmls.push("<td><a data-gene='"+geneId+"' class='link tpLoadGeneLink'>"+geneSym+"</a>");
             if (hubUrl!==null) {
                 var fullHubUrl = hubUrl+"&position="+geneSym+"&singleSearch=knownCanonical";
                 htmls.push("<a target=_blank class='link' style='margin-left: 10px; font-size:80%; color:#AAA' title='link to UCSC Genome Browser' href='"+fullHubUrl+"'>Genome</a>");
@@ -7114,10 +7462,11 @@ var cellbrowser = function() {
 
         // adult pancreas is the only dataset with an uppercase letter
         // make sure that at least at UCSC, dataset names are always lowercased.
-        // The reason is that at UCSC, we the datasetname can be part of the URL,
+        // The reason is that at UCSC, the datasetname can be part of the URL,
         // e.g. cortex-dev.cells.ucsc.edu, which the user could enter as CoRTex-dev.cells.ucsc.edu
-        // On all other servers, this is on an issue
-        if (datasetName && datasetName!=="adultPancreas" && pageAtUcsc())
+        // On all other servers, this is not an issue
+        // But never do this for subdatasets, because they often include uppercase letters
+        if (datasetName && datasetName!=="adultPancreas" && pageAtUcsc() && datasetName.indexOf("/")===-1 )
             datasetName = datasetName.toLowerCase();
         return datasetName;
     }
diff --git a/src/cbPyLib/cellbrowser/cbWeb/js/maxPlot.js b/src/cbPyLib/cellbrowser/cbWeb/js/maxPlot.js
index d63f0fa4..a5ccc990 100644
--- a/src/cbPyLib/cellbrowser/cbWeb/js/maxPlot.js
+++ b/src/cbPyLib/cellbrowser/cbWeb/js/maxPlot.js
@@ -596,6 +596,8 @@ function MaxPlot(div, top, left, width, height, args) {
      * Canvas origin is top-left, but usually plotting origin is bottom-left,
      * so also flip the Y axis. sets invisible coords to HIDCOORD
      * */
+        if (coords===null)
+            return;
         console.time("scale");
         var minX = zoomRange.minX;
         var maxX = zoomRange.maxX;
@@ -1089,8 +1091,6 @@ function MaxPlot(div, top, left, width, height, args) {
        self.calcRadius();
 
        self.coords.px = scaleCoords(self.coords.orig, borderMargin, self.port.zoomRange, self.canvas.width, self.canvas.height);
-       if (self.coords.labels!==undefined && self.coords.labels!==null)
-           self.coords.pxLabels = scaleLabels(self.coords.labels, self.port.zoomRange, borderMargin, self.canvas.width, self.canvas.height);
        if (self.coords.lines)
            self.coords.pxLines = scaleLines(self.coords.lines, self.port.zoomRange, self.canvas.width, self.canvas.height);
     }
@@ -1222,6 +1222,13 @@ function MaxPlot(div, top, left, width, height, args) {
        self.scaleData();
     };
 
+    this.setLabelCoords = function(labelCoords) {
+        /* set the label coords and return true if there were any labels before */
+        var hadLabelsBefore = self.coords.labels.length > 0;
+        self.coords.labels = labelCoords;
+        return hadLabelsBefore;
+    };
+
     this.setColorArr = function(colorArr) {
     /* set the color array, one array with one index per coordinate */
        self.col.arr = colorArr;
@@ -1306,7 +1313,7 @@ function MaxPlot(div, top, left, width, height, args) {
         console.timeEnd("draw");
 
         if (self.doDrawLabels===true && self.coords.labels!==null) {
-            self.coords.labelBbox = drawLabels(self.ctx, self.coords.pxLabels, self.canvas.width, self.canvas.height, zoomFact);
+            self.redrawLabels();
         }
 
         if (self.coords.pxLines) {
@@ -1319,6 +1326,24 @@ function MaxPlot(div, top, left, width, height, args) {
             self.childPlot.drawDots();
     };
 
+    this.redrawLabels = function() {
+        /* draw only the labels */
+        self.coords.pxLabels = scaleLabels(
+            self.coords.labels,
+            self.port.zoomRange,
+            self.port.radius,
+            self.canvas.width,
+            self.canvas.height
+        );
+        self.coords.labelBbox = drawLabels(
+            self.ctx,
+            self.coords.pxLabels,
+            self.canvas.width,
+            self.canvas.height,
+            self.port.zoomFact
+        );
+    };
+
     this.cellsAtPixel = function(x, y) {
         /* return the Ids of all cells at a particular pixel */
         var res = [];
diff --git a/src/cbPyLib/cellbrowser/cellbrowser.py b/src/cbPyLib/cellbrowser/cellbrowser.py
index 058f39c6..67f8fec1 100755
--- a/src/cbPyLib/cellbrowser/cellbrowser.py
+++ b/src/cbPyLib/cellbrowser/cellbrowser.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# this library mostly contains functions that convert tab-sep files
+# this library mostly contains functions that convert tab-sep/loom/h5ad/mtx files
 # (=single cell expression matrix and meta data) into the binary format that is read by the
 # javascript viewer cbWeb/js/cellbrowser.js and cbData.js.
 # Helper functions here allow importing data from other tools, e.g. cellranger or scanpy.
@@ -12,7 +12,7 @@
 
 import logging, sys, optparse, struct, json, os, string, shutil, gzip, re, unicodedata
 import zlib, math, operator, doctest, copy, bisect, array, glob, io, time, subprocess
-import hashlib, timeit, datetime, keyword, itertools, os.path, urllib
+import hashlib, timeit, datetime, keyword, itertools, os.path, urllib, platform
 from distutils import spawn
 from collections import namedtuple, OrderedDict
 from os.path import join, basename, dirname, isfile, isdir, relpath, abspath, getsize, getmtime, expanduser
@@ -23,10 +23,13 @@
     # python3
     from urllib.parse import urljoin
     from urllib.request import urlopen
+    HTTPERR = urllib.error.HTTPError
 except ImportError:
     # python2
     from urlparse import urljoin
     from urllib2 import urlopen
+    import urllib2
+    HTTPERR = urllib2.URLError
 
 try:
     # > python3.3
@@ -240,7 +243,7 @@ def downloadUrlBinary(remoteUrl):
             data = urlopen(remoteUrl, context=ssl._create_unverified_context()).read()
         else:
             data = urlopen(remoteUrl).read()
-    except urllib.error.HTTPError:
+    except HTTPERR:
         logging.error("Cannot download %s" % remoteUrl)
         data = None
     return data
@@ -248,6 +251,9 @@ def downloadUrlBinary(remoteUrl):
 def downloadUrlLines(url):
     " open URL, slurp in all data and return a list of the text lines "
     data = downloadUrlBinary(url)
+    if data is None:
+        errAbort("Cannot download %s" % url)
+
     if url.endswith(".gz"):
         data = gzip.decompress(data)
     lines = data.splitlines()
@@ -584,9 +590,11 @@ def cbScanpy_parseArgs():
     parser.add_option("", "--skipMarkers", dest="skipMarkers", action="store_true",
             help="do not try to calculate cluster-specific marker genes. Only useful for the rare datasets where a bug in scanpy crashes the marker gene calculation.")
 
+    parser.add_option("-f", "--matrixFormat", dest="matrixFormat", action="store",
+            help="Output matrix file format. 'mtx' or 'tsv'. default: tsv",)
+
     parser.add_option("", "--copyMatrix", dest="copyMatrix", action="store_true",
             help="Instead of reading the input matrix into scanpy and then writing it back out, just copy the input matrix. Only works if the input matrix is gzipped and in the right format and a tsv or csv file, not mtx or h5-based files.")
-
     parser.add_option("-g", "--genome", dest="genome", action="store",
             help="when reading 10X HDF5 files, the genome to read. Default is %default. Use h5ls <h5file> to show possible genomes", default="GRCh38")
 
@@ -714,16 +722,17 @@ def lineFileNextRow(inFile, headerIsRow=False, noHeaders=False) :
         # must read first line now to get number of fields
         row1 = nextEl(ifh)
         headers = ["col"+str(i) for i in range(0, len(row1))]
+        rawHeaders = headers
         Record = namedtuple('tsvCsvRec', headers)
         savedLines = [Record(*row1)]
     else:
         savedLines = []
-        headers = nextEl(ifh)
-        headers = sanitizeHeaders(headers)
+        rawHeaders = nextEl(ifh)
+        headers = sanitizeHeaders(rawHeaders)
         Record = namedtuple('tsvCsvRec', headers)
 
     if headerIsRow:
-        yield headers
+        yield rawHeaders # the calling script wants the *real* headers, e.g. "my.name", not "my_name" instead
 
     lineCount = 0
     for fields in itertools.chain(savedLines, ifh):
@@ -1055,16 +1064,6 @@ def likeEmptyString(val):
     " returns true if string is a well-known synonym of 'unknown' or 'NaN'. ported from cellbrowser.js "
     return val.strip() in emptyVals
 
-#def floatToIntList(vals):
-#    " convert a list of floats to integers, take care of -inf values "
-#    newVals = []
-#    for x in vals:
-#        if x==FLOATNAN:
-#            newVals.append(INTNAN)
-#        else:
-#            newVals.append(int(x))
-#    return newVals
-
 def itemsInOrder(valDict, keyOrder):
     """ given a dict key->val and a list of keys, return (key, val) in the order of the list
     """
@@ -1394,15 +1393,29 @@ def findMtxFiles(fname):
     else:
         matDir = dirname(fname)
 
-    mtxFname = join(matDir, "matrix.mtx.gz")
+    #mtxFname = join(matDir, "matrix.mtx.gz")
+    #if not isfile(mtxFname):
+        #errAbort("Sorry, right now, for .mtx support, the input matrix name must be matrix.mtx.gz. "
+                #"Please rename the file, adapt cellbrowser.conf and rerun the command.")
+    mtxFname = fname
+
     if not isfile(mtxFname):
-        errAbort("Sorry, right now, for .mtx support, the input matrix name must be matrix.mtx.gz. "
-                "Please rename the file, adapt cellbrowser.conf and rerun the command.")
+        errAbort("Could not find %s" % mtxFname)
+
+    prefix = ""
+    if "_" in basename(mtxFname):
+        prefix = basename(mtxFname).split("_")[0]+"_"
+        logging.debug("Basename-prefix of mtx is: %s")
 
-    genesFname = join(matDir, "genes.tsv.gz")
+    genesFname = join(matDir, prefix+"genes.tsv.gz")
     if not isfile(genesFname): # zealous cellranger 3 engineers renamed the genes file. Argh.
-        genesFname = join(matDir, "features.tsv.gz")
-    barcodeFname = join(matDir, "barcodes.tsv.gz")
+        genesFname = join(matDir, prefix+"features.tsv.gz")
+    barcodeFname = join(matDir, prefix+"barcodes.tsv.gz")
+
+    if not isfile(genesFname):
+        errAbort("Found file %s, so expected genes file %s to exist but could not find it. " % (mtxFname, genesFname))
+    if not isfile(barcodeFname):
+        errAbort("Found file %s, so expected genes file %s to exist but could not find it. " % (mtxFname, barcodeFname))
 
     logging.debug("mtx filename: %s, %s and %s" % (mtxFname, genesFname, barcodeFname))
     return mtxFname, genesFname, barcodeFname
@@ -1868,6 +1881,14 @@ def digitize_np(arr, matType):
 def maxVal(a):
     if numpyLoaded:
         return np.amax(a)
+    # if MTX old old numpy is loaded, so isNumpy is false but var is still an ndarray - weird construct to avoid syntax error on undefined "np"
+    elif 'np' in dir():
+        if isinstance(a, np.ndarray):
+            return np.amax(a)
+    elif 'numpy' in sys.modules:
+        import numpy
+        if isinstance(a, numpy.ndarray):
+            return numpy.amax(a)
     else:
         return max(a)
 
@@ -1931,7 +1952,17 @@ def exprEncode(geneDesc, exprArr, matType):
         else:
             assert(False) # internal error
 
-        exprStr = array.array(arrType, exprArr).tostring()
+        # if as too-old numpy version is loaded isNumpy is false, but the type may
+        # still be a numpy array if we loaded from MTX -> force to a list
+        if str(type(exprArr))=="<type 'numpy.ndarray'>":
+            exprArr = exprArr.tolist()[0]
+
+        # Python 3.9 removed tostring()
+        if sys.version_info >= (3, 2):
+            exprStr = array.array(arrType, exprArr).tobytes()
+        else:
+            exprStr = array.array(arrType, exprArr).tostring()
+
         minVal = min(exprArr)
 
     if isPy3:
@@ -2276,7 +2307,7 @@ def readMatrixSampleNames(fname):
     " return a list of the sample names of a matrix fname "
     if fname.endswith(".mtx.gz"):
         matrixPath = dirname(fname)
-        barcodePath = join(matrixPath, "barcodes.tsv.gz")
+        barcodePath = findMtxFiles(fname)[2]
         logging.info("Reading sample names for %s -> %s" % (matrixPath, barcodePath))
         lines = openFile(barcodePath).read().splitlines()
         ret = []
@@ -2460,7 +2491,7 @@ def runCommand(cmd, verbose=False):
     return 0
 
 def readMtxDims(fname):
-    " return the x,y dimensions of the mtx file "
+    " return the rowCount,colCount dimensions of the mtx file. Usually columns = cells "
     logging.debug("Opening MTX file %s" % fname)
     headerFound = False
     for line in openFile(fname):
@@ -2485,20 +2516,27 @@ def readMtxDims(fname):
 def checkMtx(mtxFname, geneFname, barcodeFname):
     " make sure that the dimensions of the mtx file match the sizes of the barcodes and genes files "
     logging.debug("Checking %s" % mtxFname)
-    geneCount, cellCount = readMtxDims(mtxFname)
+    rowCount, colCount = readMtxDims(mtxFname)
+    # usually columns = number of cells
     geneIds, barcodes = readGenesBarcodes(geneFname, barcodeFname)
     logging.debug("%d geneIds, %d barcodes" % (len(geneIds), len(barcodes)))
-    if len(geneIds) != geneCount:
+    if len(geneIds) == colCount:
+        errAbort("Looks like this MTX file has the genes on the columns. Please transpose the matrix, "
+        "then re-run this command.")
+
+    if len(geneIds) != rowCount:
         errAbort("The number of rows in the file %s (%d) is different from the number of lines in the file %s (%d). "
                 "The number should be identical and usually is the number of genes/features. "
                 "This suggests a problem in the way the data was exported. You may want to remove header lines. "
-                % (mtxFname, len(geneIds), geneFname, geneCount))
-    if len(barcodes) != cellCount:
+                % (mtxFname, len(geneIds), geneFname, rowCount))
+    if len(barcodes) != colCount:
         errAbort("The number of columns in the file %s is different from the number of lines in the file %s. "
                 "The number should be identical and usually is the number of genes/features. "
                 "This suggests a problem in the way the data was exported. You may want to remove header lines. "
                 % (mtxFname, barcodeFname))
 
+    return False
+
 def copyMatrixTrim(inFname, outFname, filtSampleNames, doFilter, geneToSym, outConf, matType):
     """ copy matrix and compress it. If doFilter is true: keep only the samples in filtSampleNames
     Returns the format of the matrix, "float" or "int", or None if not known
@@ -2519,14 +2557,20 @@ def copyMatrixTrim(inFname, outFname, filtSampleNames, doFilter, geneToSym, outC
 
         # XX no happy: stupid .gz filename heuristics
         if inFname.endswith(".gz"):
-            cmd = "cp \"%s\" \"%s\"" % (inFname, outFname)
+            shutil.copyfile(inFname, outFname)
         else:
-            cmd = "cat \"%s\" | gzip -c > %s" % (inFname, outFname)
-        ret = runCommand(cmd)
-
-        if ret!=0 and isfile(outFname):
-            os.remove(outFname)
-            sys.exit(1)
+            if platform.system()=="Windows":
+                # slow but quick hack for Github #229
+                tmpFname = outFname+".tmp"
+                shutil.copyfile(inFname, tmpFname)
+                runGzip(tmpFname, outFname)
+            else:
+                # faster, but works only on Linux/OSX
+                cmd = "cat \"%s\" | gzip -c > %s" % (inFname, outFname)
+                ret = runCommand(cmd)
+                if ret!=0 and isfile(outFname):
+                    os.remove(outFname)
+                    sys.exit(1)
         return matType
 
     sep = "\t"
@@ -2612,7 +2656,8 @@ def sanitizeName(name):
     newName = nonAscii.sub("", name)
     if newName!=name:
         logging.debug("Sanitized %s -> %s" % (repr(name), newName))
-    assert(len(newName)!=0)
+    if (len(newName)==0):
+        errAbort("Field name %s has only special chars?" % name)
     return newName
 
 def nonAlphaToUnderscores(name):
@@ -2683,7 +2728,12 @@ def parseMarkerTable(filename, geneToSym):
     otherColumns = defaultdict(list)
     for row in reader:
         clusterName = row[clusterIdx]
+
+        if clusterName=="":
+            errAbort("Found empty string as cluster name in cluster marker file. Please fix the marker file.")
+
         geneId = row[geneIdx]
+
         scoreVal = float(row[scoreIdx])
         otherFields = row[otherStart:otherEnd]
 
@@ -2692,6 +2742,9 @@ def parseMarkerTable(filename, geneToSym):
 
         geneSym = convIdToSym(geneToSym, geneId, printWarning=False)
 
+        if "|" in geneId:
+            geneId = geneId.split("|")[0]
+
         newRow = []
         newRow.append(geneId)
         newRow.append(geneSym)
@@ -2715,8 +2768,8 @@ def parseMarkerTable(filename, geneToSym):
     newHeaders = ["id", "symbol", headers[scoreIdx]+"|float"]
     newHeaders.extend(otherHeadersWithType)
 
-    if len(data) > 300:
-        errAbort("Your marker file has more than 300 clusters. Are you sure that this is correct? The input format is (clusterName, geneSymName, Score), is it possible that you have inversed the order of cluster and gene?")
+    if len(data) > 1000:
+        errAbort("Your marker file has more than 1000 clusters. Are you sure that this is correct? The input format is (clusterName, geneSymName, Score), is it possible that you have inversed the order of cluster and gene?")
 
     # determine if the score field is most likely a p-value, needed for sorting
     revSort = True
@@ -2753,7 +2806,6 @@ def splitMarkerTable(filename, geneToSym, outDir):
         sanNames.add(sanName)
 
         outFname = join(outDir, sanName+".tsv")
-        print("===", repr(outFname))
         logging.debug("Writing %s" % outFname)
         ofh = open(outFname, "w")
         ofh.write("\t".join(newHeaders))
@@ -2794,7 +2846,6 @@ def copyDatasetHtmls(inDir, outConf, datasetDir):
         if not isfile(inFname):
             logging.debug("%s does not exist" % inFname)
         else:
-            #copyFiles.append( (fname, "summary.html") )
             outPath = join(datasetDir, fileBase)
             logging.debug("Copying %s -> %s" % (inFname, outPath))
             shutil.copyfile(inFname, outPath)
@@ -2892,6 +2943,7 @@ def writeDatasetDesc(inDir, outConf, datasetDir, coordFiles=None, matrixFname=No
     if not isfile(confFname):
         confFname = join(inDir, "desc.conf")
 
+    # old files don't have the outConf object yet
     if "fileVersions" not in outConf:
         outConf["fileVersions"] = {}
 
@@ -2939,6 +2991,10 @@ def writeDatasetDesc(inDir, outConf, datasetDir, coordFiles=None, matrixFname=No
     if "unit" in outConf and not "unitDesc" in summInfo:
         summInfo["unitDesc"] = outConf["unit"]
 
+    # import the atachSearch attribute from cellbrowser.conf
+    if "atacSearch" in outConf and not "atacSearch" in summInfo:
+        summInfo["atacSearch"] = outConf["atacSearch"]
+
     # copy over the raw matrix file, usually this is a zip or gzip file
     if "rawMatrixFile" in summInfo:
         rawInPath = join(inDir, summInfo["rawMatrixFile"])
@@ -2990,15 +3046,8 @@ def writeDatasetDesc(inDir, outConf, datasetDir, coordFiles=None, matrixFname=No
 
     writeJson(summInfo, outPath)
 
-    #if "descMd5s" not in outConf:
-        #outConf["descMd5s"] = {}
-
-    #outConf["descMd5s"]["datasetDesc"] = md5ForFile(confFname)[:MD5LEN]
-
     # it's easier to have a single field that tells us if the desc.json is present
     if not "hasFiles" in outConf:
-        outConf["hasFiles"] = {}
-    #if ("descMd5s" in outConf) and ("datasetDesc" in outConf["descMd5s"]):
         outConf["hasFiles"] = ["datasetDesc"]
 
     logging.debug("Wrote dataset description to %s" % outPath)
@@ -3142,17 +3191,17 @@ def readHeaders(fname):
         errAbort("Could not read headers from file %s" % fname)
     return row
 
-def parseGeneInfo(geneToSym, fname):
-    """ parse a file with three columns: symbol, desc (optional), pmid (optional).
-    Return as a dict symbol -> [description, pmid] """
+def parseGeneInfo(geneToSym, fname, matrixSyms, matrixGeneIds):
+    """ parse quick genes file with three columns: symbol or geneId, desc (optional), pmid (optional).
+    Return as a dict geneId|symbol -> [description, pmid] """
     if fname is None:
         return {}
     logging.debug("Parsing %s" % fname)
-    validSyms = None
+    symToGene = None
     if geneToSym is not None:
-        validSyms = set()
+        symToGene = dict()
         for gene, sym in iterItems(geneToSym):
-            validSyms.add(sym)
+            symToGene[sym] = gene
 
     sep = sepForFile(fname)
     geneInfo = []
@@ -3164,19 +3213,56 @@ def parseGeneInfo(geneToSym, fname):
         if line.startswith("symbol"):
             continue
         row = line.rstrip("\r\n").split(sep)
-        sym = row[0]
-        if validSyms is not None and sym not in validSyms:
-            sym = geneToSym.get(sym)
-            if sym is None:
-                logging.error("'%s' is not a valid gene symbol, skipping it" % sym)
-                continue
+        geneOrSym = row[0]
+
+        # case 1: user provides both geneId and symbol. Rare.
+        # Necessary when symbol <-> geneId is not unique
+        if "|" in geneOrSym:
+            geneId, sym = geneOrSym.split("|")
+            if geneId not in matrixGeneIds:
+                errAbort("geneId %s in quickgenes file is not in expression matrix" % geneId)
+            geneStr = geneOrSym
+
+        # case 2: matrix has only symbols and user provides symbol. legacy format.
+        # store only the symbol. We could look up the geneId but that's data inference, 
+        # which we try not to do. The lookup could be wrong.
+        elif geneOrSym in matrixSyms and len(matrixGeneIds)==0:
+            geneStr = geneOrSym
+
+        # case 3: matrix has geneIds and user provides a geneId. add the symbol from our mapping
+        # that's a data inference that should not be wrong
+        elif geneOrSym in matrixGeneIds:
+            geneId = geneOrSym
+            sym = geneToSym[geneId]
+            geneStr = geneId+"|"+sym
+
+        # case 4: matrix has geneIds and user provides geneId or symbol. Store both.
+        elif geneToSym:
+            # if we have a gene symbol mapping, we can resolve the symbols to geneIds
+            # one can provide either geneIDs or symbols in the quick genes file
+            if geneOrSym in geneToSym:
+                geneId = geneOrSym
+                sym = geneToSym[geneId]
+            elif geneOrSym in symToGene:
+                sym = geneOrSym
+                geneId = symToGene[sym]
+            else:
+                errAbort("Gene %s in quickgenes file is neither a symbol nor a geneId" % geneOrSym)
+            geneStr = geneId+"|"+sym
+
+        else:
+            errAbort("Gene %s in quickgenes file is not in expr matrix and there is no geneId<->symbol mapping to resolve it to a geneId in the expression matrix" % geneOrSym)
+
+        # if we had no geneToSym, we'll check the symbol later if it's valid
+
+        info = [geneStr]
 
-        info = [sym]
         if len(row)>1:
             info.append(row[1])
         if len(row)>2:
             info.append(row[2])
         geneInfo.append(info)
+
     return geneInfo
 
 def readSampleNames(fname):
@@ -3215,7 +3301,7 @@ def guessGeneIdType(inputObj):
             matIter.open(matrixFname, usePyGzip=True)
             geneId, sym, exprArr = nextEl(matIter.iterRows())
             matIter.close()
-            geneIds = [geneId]
+            geneIds = [geneId] # only use the first geneId for the guessing
 
     geneId = geneIds[0]
 
@@ -3294,6 +3380,7 @@ def convertExprMatrix(inConf, outMatrixFname, outConf, metaSampleNames, geneToSy
     discretMatrixIndex = join(outDir, "discretMat.json")
 
     genesAreRanges = inConf.get("atacSearch")
+
     matType = matrixToBin(outMatrixFname, geneToSym, binMat, binMatIndex, discretBinMat, discretMatrixIndex, metaSampleNames, matType=matType, genesAreRanges=genesAreRanges)
 
     # these are the Javascript type names, not the python ones (they are also better to read than the Python ones)
@@ -3467,29 +3554,69 @@ def convertMarkers(inConf, outConf, geneToSym, clusterLabels, outDir):
 
     outConf["markers"] = newMarkers
 
+def areProbablyGeneIds(ids):
+    " if 90% of the identifiers start with the same letter, they are probably gene IDs, not symbols "
+    counts = Counter()
+    numCount = 0
+    for s in ids:
+        counts[s[0]]+=1
+        if s.isnumeric():
+            numCount += 1
+
+    cutoff = 0.9* len(ids)
+    if counts.most_common()[0][1] >= cutoff or numCount >= cutoff:
+        logging.debug("GeneIds in matrix are identifiers, not symbols")
+        return True
+    else:
+        logging.debug("GeneIds in matrix are symbols, not identifiers")
+        return False
+
+def readValidGenes(outDir):
+    """ the output directory contains an exprMatrix.json file that contains all valid gene symbols.
+    We're reading those here """
+    matrixJsonFname = join(outDir, "exprMatrix.json")
+    validGenes = list(readJson(matrixJsonFname))
+
+    syms = []
+    geneIds = []
+    geneToSym = {}
+    hasBoth = False
+    for g in validGenes:
+        if "|" in g:
+            parts = g.split("|")
+            geneId = parts[0]
+            sym = parts[1]
+            syms.append( sym )
+            geneIds.append(geneId)
+            hasBoth = True
+        else:
+            syms.append( g )
+
+    if not hasBoth and areProbablyGeneIds(syms):
+        geneIds = syms
+        syms = []
+
+    if len(geneToSym)==0:
+        geneToSym = None
+
+    return set(syms), set(geneIds), geneToSym
+
 def readQuickGenes(inConf, geneToSym, outDir, outConf):
     " read quick genes file and make sure that the genes in it are in the matrix "
     quickGeneFname = inConf.get("quickGenesFile")
     if quickGeneFname:
 
-        matrixJsonFname = join(outDir, "exprMatrix.json")
-        validGenes = set(readJson(matrixJsonFname))
+        matrixSyms, matrixGeneIds, geneToSymFromMatrix = readValidGenes(outDir)
+
+        # prefer the symbols from the matrix over our own symbol tables
+        if geneToSymFromMatrix is not None:
+            geneToSym = geneToSymFromMatrix
 
         fname = getAbsPath(inConf, "quickGenesFile")
-        quickGenes = parseGeneInfo(geneToSym, fname)
-
-        validQuickGenes = []
-        for quickGeneInfo in quickGenes:
-            sym = quickGeneInfo[0]
-            if "|" in sym:
-                sym = sym.split("|")[1]
-            if sym not in validGenes:
-                logging.warning("Gene %s from quickGenesFile is not in the expression matrix, skipping", sym)
-            else:
-                validQuickGenes.append(quickGeneInfo)
+        quickGenes = parseGeneInfo(geneToSym, fname, matrixSyms, matrixGeneIds)
 
-        outConf["quickGenes"] = validQuickGenes
-        logging.info("Read %d quick genes from %s, kept %d" % (len(quickGenes), fname, len(validQuickGenes)))
+        outConf["quickGenes"] = quickGenes
+        logging.info("Read %d quick genes from %s" % (len(quickGenes), fname))
 
 def getFileVersion(fname):
     data = {}
@@ -3752,11 +3879,17 @@ def matrixOrSamplesHaveChanged(datasetDir, inMatrixFname, outMatrixFname, outCon
     origSize = oldMatrixInfo ["size"]
     nowSize = getsize(inMatrixFname)
 
+    if not "fileVersions" in outConf:
+        outConf["fileVersions"] = {}
+
     # mtx has three files, so have to add their sizes to the total now
     if isMtx(inMatrixFname) and "barcodes" in lastConf["fileVersions"]:
         oldBarInfo = lastConf["fileVersions"]["barcodes"]
         oldFeatInfo = lastConf["fileVersions"]["features"]
         origSize += oldBarInfo["size"] + oldFeatInfo["size"]
+        # very old files do not have a fileVersions object
+        if not "fileVersions" in outConf:
+            outConf["fileVersions"] = {}
         outConf["fileVersions"]["barcodes"] = oldBarInfo
         outConf["fileVersions"]["features"] = oldFeatInfo
 
@@ -3771,21 +3904,14 @@ def matrixOrSamplesHaveChanged(datasetDir, inMatrixFname, outMatrixFname, outCon
             (oldMatrixInfo, nowSize))
         return True
 
-    if not "fileVersions" in outConf:
-        outConf["fileVersions"] = {}
-
     outConf["fileVersions"]["inMatrix"] = oldMatrixInfo
     outConf["fileVersions"]["outMatrix"] = lastConf["fileVersions"]["outMatrix"]
     outConf["matrixArrType"] = lastConf["matrixArrType"]
 
     metaSampleNames = readOldSampleNames(datasetDir, lastConf)
 
-    #if isfile(outMatrixFname):
     matrixSampleNames = readMatrixSampleNames(outMatrixFname)
     assert(len(matrixSampleNames)!=0)
-    #else:
-    #outFeatsName = join(datasetDir, "features.tsv.gz")
-    #matrixSampleNames = gzip.open(outFeatsName).read().splitlines()
 
     if set(metaSampleNames)!=set(matrixSampleNames):
         logging.info("meta sample samples from previous run are different from sample names in current matrix, have to reindex the matrix. Counts: %d vs. %d" % (len(metaSampleNames), len(matrixSampleNames)))
@@ -3797,6 +3923,10 @@ def matrixOrSamplesHaveChanged(datasetDir, inMatrixFname, outMatrixFname, outCon
 def readJson(fname, keepOrder=False):
     " read .json and return as a dict "
     logging.debug("parsing json file %s" % fname)
+    if not isfile(fname):
+        logging.debug("%s does not exist, returning empty dict" % fname)
+        return OrderedDict()
+
     if keepOrder:
         customdecoder = json.JSONDecoder(object_pairs_hook=OrderedDict)
         inStr = readFile(fname)
@@ -3809,8 +3939,6 @@ def readJson(fname, keepOrder=False):
 
 def orderClusters(labelCoords, outConf):
     " given the cluster label coordinates, order them by similarity "
-    #labelCoords = readJson(clusterLabelFname)
-
     # create dict with label1 -> list of (dist, label2)
     dists = defaultdict(list)
     for i, (x1, y1, label1) in enumerate(labelCoords):
@@ -3883,11 +4011,11 @@ def copyGenes(inConf, outConf, outDir):
     dbGene = inConf["atacSearch"]
     db, geneIdType = dbGene.split(".")
 
-    try:
-        inFname = getStaticFile(getGeneJsonPath(db, geneIdType))
-    except urllib.error.HTTPError as e:
+    inFname = getStaticFile(getGeneJsonPath(db, geneIdType))
+    if inFname is None:
         errAbort("The gene model file %s could not be found on the UCSC cell browser downloads server. "
-                "This means that at UCSC we do not have a prebuilt gene model file for this genome assembly "
+                "Unless your internet is down, this means that at UCSC we do not have a prebuilt "
+                "gene model file for this genome assembly "
                 "with this name. Check if this name is indeed in the format <assembly>.<geneIdName>, "
                 "then you can either contact us or use cbGenes to build the file yourself. " % dbGene)
 
@@ -3921,7 +4049,13 @@ def convertDataset(inDir, inConf, outConf, datasetDir, redo):
     inMatrixFname = getAbsPath(inConf, "exprMatrix")
     # outMetaFname/outMatrixFname are reordered & trimmed tsv versions of the matrix/meta data
     if isMtx(inMatrixFname):
-        outMatrixFname = join(datasetDir, "matrix.mtx.gz")
+        baseName = basename(inMatrixFname)
+        # our cbImportSeurat script uses the format <slotName>_matrix.mtx.gz, keep this format internally
+        if baseName.endswith("_matrix.mtx.gz") and len(baseName.split("_"))==2:
+            baseName = basename(findMtxFiles(inMatrixFname)[0])
+            outMatrixFname = join(datasetDir, baseName)
+        else:
+            outMatrixFname = join(datasetDir, "matrix.mtx.gz")
     else:
         outMatrixFname = join(datasetDir, "exprMatrix.tsv.gz")
 
@@ -3938,6 +4072,10 @@ def convertDataset(inDir, inConf, outConf, datasetDir, redo):
 
     needFilterMatrix = True
 
+    oldConfFname = join(datasetDir, "dataset.json")
+    logging.info("Loading old config from %s" % oldConfFname)
+    oldConf = readJson(oldConfFname, keepOrder=True)
+
     if doMeta or doMatrix or redo in ["meta", "matrix"]:
         # convertMeta also compares the sample IDs between meta and matrix to determine if the meta file 
         # needs reordering or trimming (=if the meta contains more cells than the matrix)
@@ -3945,9 +4083,6 @@ def convertDataset(inDir, inConf, outConf, datasetDir, redo):
     else:
         sampleNames = readSampleNames(outMetaFname)
         needFilterMatrix = False
-        oldConfFname = join(datasetDir, "dataset.json")
-        logging.info("Loading old config from %s" % oldConfFname)
-        oldConf = readJson(oldConfFname, keepOrder=True)
         outConf.update(oldConf)
 
     if doMatrix or redo=='matrix':
@@ -3973,9 +4108,18 @@ def convertDataset(inDir, inConf, outConf, datasetDir, redo):
     for tag in ["name", "shortLabel", "radius", "alpha", "priority", "tags", "sampleDesc", "geneLabel",
         "clusterField", "defColorField", "xenaPhenoId", "xenaId", "hubUrl", "showLabels", "ucscDb",
         "unit", "violinField", "visibility", "coordLabel", "lineWidth", "hideDataset", "hideDownload",
-        "metaBarWidth", "supplFiles", "body_parts", "defQuantPal", "defCatPal", "clusterPngDir"]:
+        "metaBarWidth", "supplFiles", "defQuantPal", "defCatPal", "clusterPngDir",
+        "body_parts", "organisms", "diseases", "projects" ]:
         copyConf(inConf, outConf, tag)
 
+    # for the news generator and for future logging, note when this dataset was built for the first time into this directory
+    # and also when it was built most recently
+    if "firstBuildTime" in oldConf:
+        outConf["firstBuildTime"] = oldConf["firstBuildTime"]
+    else:
+        outConf["firstBuildTime"] = datetime.datetime.now().isoformat()
+    outConf["lastBuildTime"] = datetime.datetime.now().isoformat()
+
     makeFilesTxt(outConf, datasetDir)
 
 def writeAnndataCoords(anndata, coordFields, outDir, desc):
@@ -3985,6 +4129,8 @@ def writeAnndataCoords(anndata, coordFields, outDir, desc):
     if coordFields=="all" or coordFields is None:
         coordFields = getObsmKeys(anndata)
 
+    coordFields.sort(reverse=True) # umap first, then t-sne, then PCA... sorting works to make a good order!
+
     for fieldName in coordFields:
         # examples:
         # X_draw_graph_tsne - old versions
@@ -4007,8 +4153,10 @@ def writeAnndataCoords(anndata, coordFields, outDir, desc):
         desc.append( {'file':fileBase, 'shortLabel': fullName} )
 
 def writeCellbrowserConf(name, coordsList, fname, addMarkers=True, args={}):
-    for c in name:
-        assert(c.isalnum() or c in ["-", "_"]) # only digits and letters are allowed in dataset names
+    checkDsName(name)
+    #kddfor c in name:
+        #if not (c.isalnum() or c in ["-", "_"]):
+            # only digits and letters are allowed in dataset names
 
     metaFname = args.get("meta", "meta.tsv")
     clusterField = args.get("clusterField", "Louvain Cluster")
@@ -4031,6 +4179,7 @@ def writeCellbrowserConf(name, coordsList, fname, addMarkers=True, args={}):
 enumFields=['%(clusterField)s']
 coords=%(coordStr)s
 #alpha=0.3
+#bodyParts=["embryo", "heart", "brain"]
 #radius=2
 """ % locals()
 
@@ -4054,15 +4203,91 @@ def writeCellbrowserConf(name, coordsList, fname, addMarkers=True, args={}):
     ofh.close()
     logging.info("Wrote %s" % ofh.name)
 
-def geneSeriesToStrings(geneIdSeries, indexFirst=False):
+def geneSeriesToStrings(geneIdSeries, indexFirst=False, sep="|"):
     " convert a pandas data series to a list of |-separated strings "
     if indexFirst:
         geneIdAndSyms = list(zip(geneIdSeries.index, geneIdSeries.values))
     else:
         geneIdAndSyms = list(zip(geneIdSeries.values, geneIdSeries.index))
-    genes = [str(x)+"|"+str(y) for (x,y) in geneIdAndSyms]
+    genes = [str(x)+sep+str(y) for (x,y) in geneIdAndSyms]
+    return genes
+
+def geneStringsFromVar(var, sep="|"):
+    " return a list of strings in format geneId<sep>geneSymbol "
+    if "gene_ids" in var:
+        genes = geneSeriesToStrings(var["gene_ids"], indexFirst=False, sep=sep)
+    elif "gene_symbols" in var:
+        genes = geneSeriesToStrings(var["gene_symbols"], indexFirst=True, sep=sep)
+    elif "Accession" in var:  # only seen this in the ABA Loom files
+        genes = geneSeriesToStrings(var["Accession"], indexFirst=False, sep=sep)
+    else:
+        genes = var.index.tolist()
     return genes
 
+def anndataMatrixToMtx(ad, path, useRaw=False):
+    """
+    write the ad expression matrix into a sparse mtx.gz file (matrix-market format)
+
+    to test:
+    cbImportScanpy -i /home/michi/ms_python_packages/cellBrowser/sampleData/pbmc_small/anndata.h5ad -o /tmp/cbtest
+    cbBuild -i /tmp/cbtest/cellbrowser.conf -o /tmp/cb_html/
+    """
+
+    import scipy.io
+
+    if useRaw and ad.raw is None:
+        logging.warning("The option to export raw expression data is set, but the scanpy object has no 'raw' attribute. Exporting the processed scanpy matrix. Some genes may be missing.")
+
+    if useRaw and ad.raw is not None:
+        mat = ad.raw.X
+        var = ad.raw.var
+        logging.info("Processed matrix has size (%d cells, %d genes)" % (mat.shape[0], mat.shape[1]))
+        logging.info("Using raw expression matrix")
+    else:
+        mat = ad.X
+        var = ad.var
+
+    rowCount, colCount = mat.shape
+    logging.info("Writing scanpy matrix (%d cells, %d genes) to %s in mtx.gz format" % (rowCount, colCount, path))
+
+    logging.info("Transposing matrix") # necessary, as scanpy has the samples on the rows
+    mat = mat.T
+
+    mtxfile = join(path, 'matrix.mtx')
+
+    """
+    this is stupid: if mat is dense, mmwrite screws up the header:
+    the header is supposed to contain a line with `rows cols nonzero_elements`.
+    If mat is dense, it skips the nonzero_elements, which leads to an error reading the matrix later on.
+    actually it stores it as "array" rather than matrix %%MatrixMarket matrix coordinate real general
+    """
+    dataType = "float"
+    if ad.X.dtype.kind=="i":
+        dataType = "integer"
+
+    if ~scipy.sparse.issparse(mat):
+        mat = scipy.sparse.csr_matrix(mat)
+
+    logging.info("Writing matrix to %s, type=%s" % (mtxfile, dataType)) # scanpy has the samples on the rows
+    scipy.io.mmwrite(mtxfile, mat, precision=7)
+
+    logging.info("Compressing matrix to %s.gz" % mtxfile) # necessary, as scanpy has the samples on the rows
+    # runGzip(mtxfile, mtxfile)  # this is giving me trouble with the same filename
+    with open(mtxfile,'rb') as mtx_in:
+        with gzip.open(mtxfile + '.gz','wb') as mtx_gz:
+            shutil.copyfileobj(mtx_in, mtx_gz)
+    os.remove(mtxfile)
+
+    geneLines = geneStringsFromVar(var, sep="\t")
+    genes_file = join(path, 'features.tsv.gz')
+    with gzip.open(genes_file, 'wt') as f:
+        f.write("\n".join(geneLines))
+
+    bc_file = join(path, 'barcodes.tsv.gz')
+    sampleNames = ad.obs.index.tolist()
+    with gzip.open(bc_file, 'wt') as f:
+        f.write("\n".join(sampleNames))
+
 def anndataMatrixToTsv(ad, matFname, usePandas=False, useRaw=False):
     " write ad expression matrix to .tsv file and gzip it "
     import pandas as pd
@@ -4110,14 +4335,7 @@ def anndataMatrixToTsv(ad, matFname, usePandas=False, useRaw=False):
 
         # when reading 10X files, read_h5 puts the geneIds into a separate field
         # and uses only the symbol. We prefer ENSGxxxx|<symbol> as the gene ID string
-        if "gene_ids" in var:
-            genes = geneSeriesToStrings(var["gene_ids"], indexFirst=False)
-        elif "gene_symbols" in var:
-            genes = geneSeriesToStrings(var["gene_symbols"], indexFirst=True)
-        elif "Accession" in var: # only seen this in the ABA Loom files
-            genes = geneSeriesToStrings(var["Accession"], indexFirst=False)
-        else:
-            genes = var.index.tolist()
+        genes = geneStringsFromVar(var)
 
         logging.info("Writing %d genes in total" % len(genes))
         for i, geneName in enumerate(genes):
@@ -4199,7 +4417,7 @@ def saveMarkers(adata, markerField, nb_marker, fname):
 
 def scanpyToCellbrowser(adata, path, datasetName, metaFields=None, clusterField=None,
         nb_marker=50, doDebug=False, coordFields=None, skipMatrix=False, useRaw=False,
-        skipMarkers=False, markerField='rank_genes_groups'):
+        skipMarkers=False, markerField='rank_genes_groups', matrixFormat="tsv"):
     """
     Mostly written by Lucas Seninge, lucas.seninge@etu.unistra.fr
 
@@ -4234,8 +4452,13 @@ def scanpyToCellbrowser(adata, path, datasetName, metaFields=None, clusterField=
     import anndata
 
     if not skipMatrix:
-        matFname = join(path, 'exprMatrix.tsv.gz')
-        anndataMatrixToTsv(adata, matFname, useRaw=useRaw)
+        if matrixFormat=="tsv" or matrixFormat is None:
+            matFname = join(path, 'exprMatrix.tsv.gz')
+            anndataMatrixToTsv(adata, matFname, useRaw=useRaw)
+        elif matrixFormat=="mtx":
+            anndataMatrixToMtx(adata, path, useRaw=useRaw)
+        else:
+            assert(False) # invalid value for 'matrixFormat'
 
     coordDescs = []
 
@@ -4259,6 +4482,14 @@ def scanpyToCellbrowser(adata, path, datasetName, metaFields=None, clusterField=
                     if fieldName in adata.obs.columns:
                         logging.info("Found field '%s', using it as the cell cluster field." % fieldName)
                         foundField = fieldName
+                if foundField is None:
+                    logging.info("No exact match found, searching for first field that contains 'luste' or 'ype'")
+                    for fieldName in adata.obs.columns:
+                        if "luste" in fieldName or "ype" in fieldName:
+                            logging.info("Found field %s" % fieldName)
+                            foundField = fieldName
+                            break
+
                 if foundField is None:
                     errAbort("Could not find field '%s' in the scanpy object. To make a cell browser, you should have a "
                     " field like 'cluster' or "
@@ -4585,6 +4816,7 @@ def build(confFnames, outDir, port=None, doDebug=False, devMode=False, redo=None
         datasets.append(dsName)
 
         outConf = OrderedDict()
+        outConf["fileVersions"] = dict()
 
         todoConfigs = set()
 
@@ -4595,8 +4827,8 @@ def build(confFnames, outDir, port=None, doDebug=False, devMode=False, redo=None
             logging.debug("Adding %s" % inPath)
             todoConfigs.add(inPath)
         else:
-            convertDataset(inDir, inConf, outConf, datasetDir, redo)
             copyGenes(inConf, outConf, outDir)
+            convertDataset(inDir, inConf, outConf, datasetDir, redo)
 
         # find all parent cellbrowser.conf-files
         if dataRoot is None:
@@ -4606,7 +4838,7 @@ def build(confFnames, outDir, port=None, doDebug=False, devMode=False, redo=None
             if not "fileVersions" in outConf:
                 outConf = loadConfig(inConfFname, ignoreName=True)
                 outConf["fileVersions"] = {}
-            parentFnames, fullPath, parentInfos = findParentConfigs(inConfFname, dataRoot, outConf["name"])
+            parentFnames, fullPath, parentInfos = findParentConfigs(inConfFname, dataRoot, dsName)
             todoConfigs.update(parentFnames)
             outConf["parents"] = parentInfos
             if "name" in outConf:
@@ -4788,15 +5020,49 @@ def cbBuildCli():
         traceback.print_exception(*exc_info)
         sys.exit(1)
 
-def readMatrixAnndata(matrixFname, samplesOnRows=False, genome="hg38"):
-    " read an expression matrix and return an adata object. Supports .mtx, .h5 and .tsv (not .tsv.gz) "
+def importLoom(inFname, reqCoords=False):
+    " load a loom file with anndata and if reqCoords is true, fix up the obsm attributes "
+    import pandas as pd
+    import anndata
+    ad = anndata.read_loom(inFname)
+
+    if not reqCoords:
+        return ad
+
+    coordKeyList = (["_tSNE1", "_tSNE2"], ["_X", "_Y"], ["UMAP1","UMAP2"], \
+            ['Main_cluster_umap_1', 'Main_cluster_umap_2'])
+    obsKeys = getObsKeys(ad)
+    foundCoords = False
+    for coordKeys in coordKeyList:
+        if coordKeys[0] in obsKeys and coordKeys[1] in obsKeys:
+            logging.debug("Found %s in anndata.obs, moving these fields into obsm" % repr(coordKeys))
+            newObj = pd.concat([ad.obs[coordKeys[0]], ad.obs[coordKeys[1]]], axis=1)
+            ad.obsm["tsne"] = newObj
+            del ad.obs[coordKeys[0]]
+            del ad.obs[coordKeys[1]]
+            foundCoords = True
+            break
+
+    if not foundCoords:
+        logging.warn("Did not find any keys like %s in anndata.obs, cannot import coordinates" % repr(coordKeyList))
+
+    return ad
+
+def readMatrixAnndata(matrixFname, samplesOnRows=False, genome="hg38", reqCoords=False):
+    """ read an expression matrix and return an adata object. Supports .mtx.h5 and .tsv (not .tsv.gz)
+    If reqCoords is True, will try to find and convert dim. reduc. coordinates in a file (loom).
+    """
     sc = importScanpy()
 
     if matrixFname.endswith(".mtx.gz"):
         errAbort("For cellranger3-style .mtx files, please specify the directory, not the .mtx.gz file name")
 
     if matrixFname.endswith(".h5ad"):
-        adata = sc.read(matrixFname, cache=False)
+        logging.info("File name ends with .h5ad: Loading %s using sc.read" % matrixFname)
+        adata = sc.read_h5ad(matrixFname)
+
+    elif matrixFname.endswith(".loom"):
+        adata = importLoom(matrixFname, reqCoords=reqCoords)
 
     elif isMtx(matrixFname):
         import pandas as pd
@@ -4827,7 +5093,7 @@ def readMatrixAnndata(matrixFname, samplesOnRows=False, genome="hg38"):
         adata = sc.read_10x_h5(matrixFname, genome=genome)
 
     else:
-        logging.info("Loading expression matrix: scanpy-supported format, like loom, tab-separated, etc.")
+        logging.info("Loading expression matrix: Filename does not end with h5ad/mtx/loom, so trying sc.read for any other scanpy-supported format, like tab-separated, etc.")
         if matrixFname.endswith(".loom"):
             logging.info("This is a loom file: activating --samplesOnRows")
             samplesOnRows = True
@@ -5049,7 +5315,7 @@ def summarizeDatasets(datasets):
                 #summDs[t] = ds[t]
 
         # these are copied and checked for the correct type
-        for optListTag in ["tags", "hasFiles", "body_parts"]:
+        for optListTag in ["tags", "hasFiles", "body_parts", "diseases", "organisms", "projects"]:
             if optListTag in ds:
                 if (type(ds[optListTag])==type([])): # has to be a list
                     summDs[optListTag] = ds[optListTag]
@@ -5475,11 +5741,19 @@ def cbScanpy(matrixFname, inMeta, inCluster, confFname, figDir, logFname):
 
     pipeLog("After filtering: Data has %d samples/observations and %d genes/variables" % (len(adata.obs), len(adata.var)))
 
+    sampleCountPostFilter = len(adata.obs)
+
     if len(list(adata.obs_names))==0:
         errAbort("No cells left after filtering. Consider lowering the minGenes/minCells cutoffs in scanpy.conf")
     if len(list(adata.var_names))==0:
         errAbort("No genes left after filtering. Consider lowering the minGenes/minCells cutoffs in scanpy.conf")
 
+    removedRatio = 1.0 - (float(sampleCountPostFilter) / float(sampleCount))
+    if removedRatio > 0.5:
+        logging.warn("!! The filtering removed more than 50% of cells - are you sure this is intentional? Consider lowering minGenes/minCells in scanpy.conf")
+    if removedRatio > 0.9:
+        errAbort("More than 90% of cells were filtered out, this is unlikely to make sense")
+
     if not "n_counts" in list(adata.obs.columns.values):
         logging.debug("Adding obs.n_counts")
         adata.obs['n_counts'] = np.sum(adata.X, axis=1)
@@ -5557,7 +5831,7 @@ def cbScanpy(matrixFname, inMeta, inCluster, confFname, figDir, logFname):
             sc.pp.highly_variable_genes(adata, min_mean=minMean, max_mean=maxMean, min_disp=minDisp)
             sc.pl.highly_variable_genes(adata)
         except:
-            if doExp==False:
+            if conf.get("doExp")!=True:
                 pipeLog("An error occurred when finding highly variable genes. This may be due to an input matrix file that is log'ed. Set doExp=True in the cbScanpy config file to undo the logging when the matrix is loaded and rerun the cbScanpy command")
             raise
 
@@ -5807,13 +6081,18 @@ def generateQuickGenes(outDir):
 
 def checkDsName(datasetName):
     " make sure that datasetName contains only ASCII chars and - or _, errAbort if not "
+    msg = "The dataset name '%s' is not a valid name . " \
+        "Valid names can only contain letters, numbers and dashes. " \
+        "If this dataset was imported from a h5ad or a Seurat object, which can provide dataset names, " \
+        "you can use the option -n of the command line tools to override the name. " % datasetName
+
     if datasetName.startswith("-"):
-        errAbort("dataset name cannot start with a dash. (forgot to supply an argument for -n?)")
+        errAbort(msg+"Dataset name cannot start with a dash. (forgot to supply an argument for -n?)")
     if "/" in datasetName:
-        errAbort("dataset name cannot contain slashes, these are reserved for collections")
+        errAbort(msg+"Dataset name cannot contain slashes, these are reserved for collections")
     match = re.match("^[a-z0-9A-Z-_]*$", datasetName)
     if match is None:
-        errAbort("dataset name can only contain lower or uppercase letters or dash or underscore")
+        errAbort(msg+"Dataset name can only contain lower or uppercase letters or dash or underscore")
 
 def importScanpy():
     try:
@@ -5831,6 +6110,14 @@ def importScanpy():
         sys.exit(1)
     return sc
 
+def getMatrixFormat(options):
+    " return matrix format, either from options object or from config file "
+    matrixFormat = options.matrixFormat
+    # command line has priority
+    if matrixFormat is None:
+        matrixFormat = getConfig("matrixFormat", matrixFormat)
+    return matrixFormat
+
 def cbScanpyCli():
     " command line interface for cbScanpy "
     mustBePython3()
@@ -5858,6 +6145,8 @@ def cbScanpyCli():
         datasetName = basename(dirname(abspath(outDir)))
         logging.info("no dataset name provided, using '%s' as dataset name" % datasetName)
 
+    matrixFormat = getMatrixFormat(options)
+
     checkDsName(datasetName)
 
     if copyMatrix and not matrixFname.endswith(".gz"):
@@ -5881,7 +6170,8 @@ def cbScanpyCli():
     adata.write(adFname)
 
     scanpyToCellbrowser(adata, outDir, datasetName=datasetName, skipMarkers=skipMarkers,
-            clusterField=inCluster, skipMatrix=(copyMatrix or skipMatrix), useRaw=True)
+            clusterField=inCluster, skipMatrix=(copyMatrix or skipMatrix), matrixFormat=matrixFormat,
+            useRaw=True)
 
     if copyMatrix:
         outMatrixFname = join(outDir, "exprMatrix.tsv.gz")
diff --git a/src/cbPyLib/cellbrowser/convert.py b/src/cbPyLib/cellbrowser/convert.py
index 319126ff..af34f04a 100644
--- a/src/cbPyLib/cellbrowser/convert.py
+++ b/src/cbPyLib/cellbrowser/convert.py
@@ -7,7 +7,7 @@
 from .cellbrowser import runGzip, openFile, errAbort, setDebug, moveOrGzip, makeDir, iterItems
 from .cellbrowser import mtxToTsvGz, writeCellbrowserConf, getAllFields, readMatrixAnndata
 from .cellbrowser import anndataMatrixToTsv, loadConfig, sanitizeName, lineFileNextRow, scanpyToCellbrowser, build
-from .cellbrowser import generateHtmls, getObsKeys, renameFile
+from .cellbrowser import generateHtmls, getObsKeys, renameFile, getMatrixFormat
 
 from os.path import join, basename, dirname, isfile, isdir, relpath, abspath, getsize, getmtime, expanduser
 
@@ -497,7 +497,8 @@ def cbCellrangerCli_parseArgs(showHelp=False):
 
 def cbImportScanpy_parseArgs(showHelp=False):
     " setup logging, parse command line arguments and options. -h shows auto-generated help page "
-    parser = optparse.OptionParser("""usage: %prog [options] inFilename outDir datasetName - convert Scanpy AnnData object to cellbrowser. inFilename can be an .h5ad or .loom file.
+    parser = optparse.OptionParser("""usage: %prog [options] -i inFilename -o outDir - convert Scanpy AnnData object to cellbrowser. inFilename can be an .h5ad or .loom file. 
+    Exports raw.X by default, or .X alternatively, see --proc. Exports all meta data. Writes .tsv by default, or alternatively .mtx.gz
 
     Example:
     - %prog -i pbmc3k.h5ad -o pbmc3kScanpy - convert pbmc3k to directory with tab-separated files
@@ -516,7 +517,7 @@ def cbImportScanpy_parseArgs(showHelp=False):
         help="Output directory. Required parameter")
 
     parser.add_option("-n", "--name", dest="datasetName", action="store",
-        help="Dataset name for generated cellbrowser.conf. If not specified, the last component of -o will be used.")
+        help="Dataset name for generated cellbrowser.conf. If not specified, the last component of -o will be used")
 
     parser.add_option("", "--htmlDir", dest="htmlDir", action="store",
         help="do not only convert to tab-sep files but also run cbBuild to"
@@ -531,6 +532,9 @@ def cbImportScanpy_parseArgs(showHelp=False):
     parser.add_option("", "--clusterField", dest="clusterField", action="store",
             help="if no marker genes are present, use this field to calculate them. Default is to try a list of common field names, like 'Cluster' or 'louvain' and a few others")
 
+    parser.add_option("-f", "--matrixFormat", dest="matrixFormat", action="store",
+            help="Output matrix file format. 'mtx' or 'tsv'. default: tsv",)
+
     parser.add_option("-m", "--skipMatrix", dest="skipMatrix", action="store_true",
         help="do not convert the matrix, saves time if the same one has been exported before to the "
         "same directory")
@@ -547,29 +551,6 @@ def cbImportScanpy_parseArgs(showHelp=False):
     setDebug(options.debug)
     return args, options
 
-def importLoom(inFname):
-    " load a loom file with anndata and fix up the obsm attributes "
-    import pandas as pd
-    import anndata
-    ad = anndata.read_loom(inFname)
-
-    coordKeyList = (["_tSNE1", "_tSNE2"], ["_X", "_Y"], ["UMAP1","UMAP2"], ['Main_cluster_umap_1', 'Main_cluster_umap_2'])
-    obsKeys = getObsKeys(ad)
-    foundCoords = False
-    for coordKeys in coordKeyList:
-        if coordKeys[0] in obsKeys and coordKeys[1] in obsKeys:
-            logging.debug("Found %s in anndata.obs, moving these fields into obsm" % repr(coordKeys))
-            newObj = pd.concat([ad.obs[coordKeys[0]], ad.obs[coordKeys[1]]], axis=1)
-            ad.obsm["tsne"] = newObj
-            del ad.obs[coordKeys[0]]
-            del ad.obs[coordKeys[1]]
-            foundCoords = True
-            break
-
-    if not foundCoords:
-        logging.warn("Did not find any keys like %s in anndata.obs, cannot import coordinates" % repr(coordKeyList))
-    return ad
-
 def cbImportScanpyCli():
     " convert h5ad to directory "
     args, options = cbImportScanpy_parseArgs()
@@ -585,20 +566,17 @@ def cbImportScanpyCli():
 
     datasetName = options.datasetName
     if datasetName is None:
-        datasetName = basename(outDir.rstrip("/"))
+        datasetName = basename(abspath(outDir))
 
     markerField = options.markerField
     clusterField = options.clusterField
     skipMarkers = options.skipMarkers
+    matrixFormat = getMatrixFormat(options)
 
-    if inFname.endswith(".loom"):
-        ad = importLoom(inFname)
-    else:
-        import anndata
-        ad = anndata.read_h5ad(inFname)
+    ad = readMatrixAnndata(inFname, reqCoords=True)
 
     scanpyToCellbrowser(ad, outDir, datasetName, skipMatrix=options.skipMatrix, useRaw=(not options.useProc),
-            markerField=markerField, clusterField=clusterField, skipMarkers=skipMarkers)
+            markerField=markerField, clusterField=clusterField, skipMarkers=skipMarkers, matrixFormat=matrixFormat)
     generateHtmls(datasetName, outDir)
 
     if options.port and not options.htmlDir:
diff --git a/src/cbPyLib/cellbrowser/genes.py b/src/cbPyLib/cellbrowser/genes.py
index 57b75f7b..b684f40a 100755
--- a/src/cbPyLib/cellbrowser/genes.py
+++ b/src/cbPyLib/cellbrowser/genes.py
@@ -170,7 +170,9 @@ def iterGencodePairs(release, doTransGene=False):
     url = "https://hgdownload.cse.ucsc.edu/goldenPath/%s/database/wgEncodeGencodeAttrsV%s.txt.gz" %  (db, release)
     logging.info("Downloading %s" % url)
     doneIds = set()
-    for line in downloadUrlLines(url):
+
+    lines = downloadUrlLines(url)
+    for line in lines:
         row = line.rstrip("\n").split("\t")
 
         if doTransGene:
@@ -428,6 +430,7 @@ def bedToJson(db, geneIdType, jsonFname):
     ofh.write(outs)
     ofh.close()
     logging.info("Wrote %s" % jsonFname)
+    logging.info("If this is a new .json file and you are on hgwdev, copy it now to /usr/local/apache/htdocs-cells/downloads/cellbrowserData/genes/ and note this directory for the dataset release push to the RR. The reason is that users may want to cbBuild using this gene transcript set and that is easier if we provide the .json file")
 
     #fileInfo[code] = {"label":label, "file" : jsonFname, "md5" :md5}
 
diff --git a/src/cbPyLib/cellbrowser/sampleConfig/cellbrowser.conf b/src/cbPyLib/cellbrowser/sampleConfig/cellbrowser.conf
index fe5bb75c..2386975a 100644
--- a/src/cbPyLib/cellbrowser/sampleConfig/cellbrowser.conf
+++ b/src/cbPyLib/cellbrowser/sampleConfig/cellbrowser.conf
@@ -67,6 +67,13 @@ labelField="WGCNAcluster"
 
 # --------- OPTIONAL SETTINGS --------------
 
+# The settings below are used to create filters on a Cell Browser instance (e.g. cells.ucsc.edu).
+# Note, all filter values can be a list (e.g. body_parts = ["brain","cortex"])
+body_parts = [""] # body_parts = ["heart"]
+organisms = [""] # organisms = ["Human (H. sapiens)", "Mouse (M. musculus)"]
+diseases = [""] # diseases = ["Healthy"]
+projects = [""] # projects = ["GTEx", "Human Cell Atlas", "hca"]
+
 # genes that are highlighted in your paper can be pre-loaded and are shown as a clickable table on the left
 # this is optional but we highly recommend that you define at least 2-3 quick genes, it makes the browser a lot
 # more intuitive for users
@@ -136,3 +143,8 @@ matrixType='auto'
 # For now, you will have to copy these images to the source destination htdocs directory manually
 # right now, only brain-lipids/all-lipids is using this
 # clusterPngDir = "clusterImgs"
+
+# Enter organs/organ parts here in a comma-separated list. Organs entered will show up as filter options for dataset list.
+# Really only useful if you have a cell browser with multiple datasets covering multiple organs (e.g. cells.ucsc.edu)
+# If no datasets have 'body_parts setting, filter won't be displayed. 
+# body_parts=["eye","cornea"]
diff --git a/src/cbPyLib/cellbrowser/seurat.py b/src/cbPyLib/cellbrowser/seurat.py
index 1a35741b..8f8bbb79 100644
--- a/src/cbPyLib/cellbrowser/seurat.py
+++ b/src/cbPyLib/cellbrowser/seurat.py
@@ -97,10 +97,19 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
         if version is None:
             errAbort("Could not find or install Seurat")
 
-    if version.split(".")[0]=="3":
+    isSeurat2 = False
+    isSeurat3 = False
+    isSeurat4 = False
+
+    verStr = version.split(".")[0]
+    if verStr=="2":
+        isSeurat2 = True
+    elif verStr=="3":
         isSeurat3 = True
+    elif verStr=="4":
+        isSeurat4 = True
     else:
-        isSeurat3 = False
+        assert(False) # unsupported Seurat version
 
     cmds = []
     # try to install Seurat if not already installed
@@ -153,20 +162,22 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     cmds.append('print("Seurat: Setup")')
     if isSeurat3:
         cmds.append('sobj <- CreateSeuratObject(mat)')
+    elif isSeurat4:
+        cmds.append('sobj <- CreateSeuratObject(counts = mat, min.cells=%d, min.genes=%d)' % (minCells, minGenes))
     else:
         cmds.append('sobj <- CreateSeuratObject(raw.data = mat, min.cells=%d, min.genes=%d)' % (minCells, minGenes))
     cmds.append('sobj') # print size of the matrix
 
     # find mito genes, mito-%, and create plots for it
     # XX - ENSG names?
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('sobj[["percent.mt"]] <- PercentageFeatureSet(sobj, pattern = "^MT-")')
     else:
         cmds.append('mito.genes <- grep(pattern = "^MT-", x = rownames(x = sobj@data), value = TRUE)')
         cmds.append('percent.mito <- Matrix::colSums(sobj@raw.data[mito.genes, ])/Matrix::colSums(sobj@raw.data)')
         cmds.append('sobj <- AddMetaData(object = sobj, metadata = percent.mito, col.name = "percent.mito")')
 
-    if not isSeurat3:
+    if isSeurat2:
         cmds.append('VlnPlot(object = sobj, features.plot = c("nGene", "nUMI", "percent.mito"), nCol = 3)')
         cmds.append('par(mfrow = c(1, 2))')
         cmds.append('GenePlot(object = sobj, gene1 = "nUMI", gene2 = "percent.mito")')
@@ -181,7 +192,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     cmds.append('print("Removing cells with more than %f percent of mitochondrial genes")' % (100*mitoMax))
     cmds.append('print("Keeping only cells with a geneCount %d-%d")' % (minGenes, maxGenes))
 
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('sobj <- subset(sobj, subset = nFeature_RNA > %(minGenes)d & nFeature_RNA < %(maxGenes)d & percent.mt < %(mitoMax)f)' % locals())
     else:
         cmds.append('sobj <- FilterCells(object = sobj, subset.names = c("nGene", "percent.mito"), low.thresholds = c(%(minGenes)d, -Inf), high.thresholds = c(%(maxGenes)d, %(mitoMax)f))' % locals())
@@ -190,7 +201,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     # do the log
     cmds.append('sobj <- NormalizeData(object = sobj, normalization.method = "LogNormalize", scale.factor = 10000)')
 
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('sobj <- FindVariableFeatures(sobj, selection.method = "vst", nfeatures = 2000)')
     else:
         # find most variable genes and gate on it
@@ -202,7 +213,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     cmds.append('sobj') # print size of the matrix
 
     # scale
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('sobj <- ScaleData(object = sobj)')
     else:
         cmds.append('sobj <- ScaleData(object = sobj, vars.to.regress = c("nUMI", "percent.mito"))')
@@ -218,7 +229,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
         #cmds.append('sobj <- JackStraw(object = sobj, num.replicate = 100, display.progress = FALSE)')
         #cmds.append('JackStrawPlot(object = sobj, PCs = 1:12)')
         #cmds.append('sobj <- RunPCA(object = sobj, pcs.compute = 50, pc.genes = sobj@var.genes, do.print = FALSE, pcs.print = 1:5, genes.print = 5)')
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('sobj <- RunPCA(sobj, npcs = %d)' % pcCountConfig)
         cmds.append('VizDimLoadings(sobj, dims = 1:2, reduction = "pca")')
         cmds.append('VizDimLoadings(sobj, dims = 2:3, reduction = "pca")')
@@ -233,7 +244,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     louvainRes = conf.get("louvainRes", 0.6)
     cmds.append('print("Finding clusters with resolution %f")' % louvainRes)
 
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('sobj <- FindNeighbors(sobj, dims=%d)' % (min(10, pcCountConfig)))
         cmds.append('sobj <- FindClusters(sobj, resolution = %f)' % louvainRes)
     else:
@@ -244,7 +255,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     perplexity = str(conf.get("perplexity", 30))
 
     doUmap = conf.get("doUmap", False)
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append("sobj <- RunTSNE(sobj, perplexity=%s)" % perplexity)
         if doUmap:
             cmds.append("sobj <- RunUMAP(sobj, dims=1:%s)" % str(pcCountConfig))
@@ -255,7 +266,7 @@ def writeCbSeuratScript(conf, inData, tsnePath, clusterPath, markerPath, rdsPath
     minMarkerPerc = conf.get("minMarkerPerc", 0.25)
     cmds.append('print("Finding markers")')
 
-    if isSeurat3:
+    if isSeurat3 or isSeurat4:
         cmds.append('all.markers <- FindAllMarkers(object = sobj)')
         cmds.append('sobj@misc[["markers"]] <- all.markers')
 
@@ -327,7 +338,7 @@ def cbSeuratCli():
 
     writeCellbrowserConf(datasetName, coords, cbConfPath, args=confArgs)
 
-    generateHtmls(datasetName, outDir, desc = {"supplFiles": [{"label":"Seurat RDS", "file":inMatrix}]})
+    generateHtmls(datasetName, outDir, desc = {"supplFiles": [{"label":"Seurat RDS", "file":basename(rdsPath)}]})
     copyPkgFile("sampleConfig/desc.conf", outDir)
 
 def cbImportSeurat_parseArgs(showHelp=False):
diff --git a/ucsc/addTags b/ucsc/addTags
new file mode 100755
index 00000000..c73c6f91
--- /dev/null
+++ b/ucsc/addTags
@@ -0,0 +1,106 @@
+#! /usr/bin/env python3
+
+import sys,argparse
+from collections import OrderedDict
+from os.path import join, basename, dirname
+from pathlib import Path
+
+# These two functions copied and modified from cellbrowser.py
+def readLines(lines, fname):
+    " recursively read lines from fname, understands lines like #include 'filename.conf' "
+    for line in open(fname):
+        line = line.rstrip("\r\n")
+        if line.startswith("#include"):
+            newFname = splitOnce(line, " ")[1]
+            newFname = newFname.strip('"').strip("'")
+            lines.extend(readLines(lines, newFname))
+        else:
+            lines.append(line)
+    return lines
+
+def loadConfig(fname):
+    """ parse python in fname and return variables as dictionary.
+    add the directory of fname to the dict as 'inDir'.
+    """
+
+    conf = OrderedDict()
+
+    g = {}
+    g["fileBase"] = basename(fname).split('.')[0]
+    g["dirName"] = basename(dirname(fname))
+
+    lines = readLines([], fname)
+    exec("\n".join(lines), g, conf)
+
+    return conf
+
+# Set up script arguments
+parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    description="Add/update tag values for a list of datasets")
+parser.add_argument('datasetFile', type=str, help='file containing list of datasets in first column \
+and any tags to add/update in subsequent columns. Header should contain tag names that should match \
+those in the example cellbrowser.conf, e.g. body_parts.')
+parser.add_argument("-u","--update", action='store_true',
+    help='if a tag already exists in the cellbrowser.conf, update it with value from datasetsFile')
+args = parser.parse_args()
+
+# From https://stackoverflow.com/questions/541390/extracting-extension-from-filename-in-python
+dFtype = Path(args.datasetFile).suffix
+
+if dFtype == ".csv":
+    dsep = ","
+elif dFtype == ".tsv":
+    dsep = "\t"
+else:
+    print("datasetFile must have tsv or csv file extension")
+    exit(1)
+
+dfh = open(args.datasetFile)
+
+# Peel off header so we can know which tags we're working with
+header = dfh.readline().rstrip().split(dsep)
+
+# Now process rest of datasetFile
+for line in dfh:
+    splitLine = line.rstrip().split(dsep)
+    dname = splitLine[0]
+    confName = join("/hive/data/inside/cells/datasets",dname,"cellbrowser.conf")
+    # Turn conf into dict
+    conf = loadConfig(confName)
+
+    # go through each tag in the header
+    for tag in header:
+        # Skip first column
+        if tag != "dataset":
+            # Get index of tag in the header so that we can use position to get values
+            # for the current tag of our current dataset later
+            n = header.index(tag)
+
+            # Case where we're dealing with a new tag
+            if tag not in conf.keys():
+                # Turn vals into a list
+                vals = splitLine[n].strip().split(' ')
+                # Append to file
+                with open(confName,"a") as f:
+                    f.write(tag + "=" + str(vals) + "\n")
+            # Case where tag already exists in conf file, so we're updating it
+            if args.update:
+                # empty string to hold outConf if we're dealing with updates to current tags
+                # doing it this way should keep the formatting of the lines we're not changing intact
+                outConf = ""
+                # go through line by line
+                for tagLine in open(confName):
+                    splitTag = tagLine.strip().split("=")
+                    confTag = splitTag[0]
+                    # find those tags we've referenced in the header
+                    if confTag == tag:
+                        # take the values from datasetFile and add to output string
+                        vals = splitLine[n].strip().split(' ')
+                        outConf += tag + "=" + str(vals) + "\n"
+                    # Just add other lines to the output string
+                    else:
+                        outConf += tagLine
+                # Write output string to our conf file
+                with open(confName,"w") as f:
+                    f.write(outConf)
diff --git a/ucsc/colorExporter b/ucsc/colorExporter
new file mode 100755
index 00000000..95320213
--- /dev/null
+++ b/ucsc/colorExporter
@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+import anndata as ad
+import pandas as pd
+import csv, sys, argparse
+
+def main():
+	parser=argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter,description="Fetch colors from h5ad")
+	parser.add_argument('-i','--input_h5ad', help='input h5ad file')
+	parser.add_argument('-o','--output_file', help='output filename in .tsv format')
+	parser.add_argument('-c', '--color_names', help='tab-separated file of metadata keys and corresponding color values. Only needed if metadata field names do not match color values in uns.')
+	args=parser.parse_args()
+
+	# Print help message if no arguments are supplied
+	if len(sys.argv) == 1:
+		parser.print_help(sys.stderr)
+		sys.exit(1)
+
+
+	a=ad.read_h5ad(args.input_h5ad)
+	if args.color_names:
+		colornames={}
+		with open(args.color_names) as colorfile:
+			for line in colorfile:
+				line=line.strip().split('\t')
+				colornames[line[0]]=line[1]
+
+
+	with open(args.output_file,"w") as csv_file:
+		writer = csv.writer(csv_file, delimiter='\t')
+		for metavalue in a.obs.keys():
+			if str(metavalue+'_colors') in a.uns.keys():
+				for key,value in {**dict(zip(a.obs[metavalue].cat.categories,a.uns[str(metavalue+'_colors')]))}.items():
+					writer.writerow([key,value])
+		if args.color_names:
+			for key in colornames:
+				for key,value in {**dict(zip(a.obs[key].cat.categories,a.uns[str(colornames[key])]))}.items():
+					writer.writerow([key,value])
+
+if __name__ == "__main__":
+	main()
diff --git a/ucsc/columnCorrection b/ucsc/columnCorrection
index a7eaf23a..e4c0707d 100755
--- a/ucsc/columnCorrection
+++ b/ucsc/columnCorrection
@@ -28,10 +28,10 @@ else:
 if repFtype == ".csv":
     rsep = ","
 elif repFtype == ".tsv":
+    rsep = "\t"
 else:
     print("replaceFile must have tsv or csv file extension")
     exit(1)
-    rsep = "\t"
 
 # Build a dict of old -> new values from file
 rfh=open(args.replaceFile, "r")
diff --git a/ucsc/datasetDiffs b/ucsc/datasetDiffs
index ec072ee3..f46554f7 100755
--- a/ucsc/datasetDiffs
+++ b/ucsc/datasetDiffs
@@ -1,31 +1,78 @@
 #! /usr/bin/env python3
 
-import json, sys, operator, argparse
+import json, sys, operator, argparse, os, urllib.request, subprocess
 
-def print_diffs(key, beta_val, test_val):
+def print_diffs(key, mach1_name, mach1_val, mach2_name, mach2_val):
     """Print out values for cells-beta and cells-test"""
-    print(key, "\n", "cells-beta:", beta_val, "\n", "cells-test:", test_val)
+    print(key, "\n", mach1_name + ": ", mach1_val, "\n", mach2_name + ": ", mach2_val)
 
-def process_dict(betaDict, testDict):
+def process_dict(mach1_name, mach1_dict, mach2_name, mach2_dict):
     """Processes a dictionary, printing out differenting values if needed.
        Or continuing to process if it finds another dictionary."""
-    for key in betaDict.keys():
-        beta_val = betaDict[key]
-        test_val = testDict[key]
-        if isinstance(beta_val, dict):
-            process_dict(beta_val, test_val)
+    for key in mach1_dict.keys():
+        mach1_val = mach1_dict[key]
+        mach2_val = mach2_dict[key]
+        if isinstance(mach1_val, dict):
+            process_dict(mach1_name, mach1_val, mach2_name, mach2_val)
         else:
-            if beta_val != test_val:
-                print_diffs(key, beta_val, test_val)
+            if mach1_val != mach2_val:
+                print_diffs(key, mach1_name, mach1_val, mach2_name, mach2_val)
+
+def compare_machines(mach1_dict, mach1_name, mach2_dict, mach2_name, verbose=False):
+    """
+    """
+    print(f'\nDatasets with diffs between {mach1_name} + {mach2_name}:')
+    for dataset in mach2_dict:
+        # Check if the overall dicts are the same for a dataset on test/beta
+        # If not, we'll go through and print out only those values that are different
+        try:
+            if not operator.eq(mach1_dict[dataset], mach2_dict[dataset]):
+                # Print dataset name
+                print("\n" + dataset)
+
+                # Script has option to print only names of datasets that have diffs
+                # This whole chunk gets skipped if that's the case
+                if verbose:
+                    for key in mach2_dict[dataset].keys():
+                        # Check if current value is a dict and if so, process that a certain way
+                        if isinstance(mach2_dict[dataset][key], dict):
+                            process_dict(mach2_name, mach2_dict[dataset][key], mach1_name, mach1_dict[dataset][key])
+                        # Otherwise check to see if values for test/beta are different
+                        # and print only the diffs
+                        else:
+                            try:
+                                val1 = mach1_dict[dataset][key]
+                                val2 = mach2_dict[dataset][key]
+                                if val1 != val2:
+                                    # The value for this keys if often large and bloats the results
+                                    # making it difficult to actually see the diffs. Only print a
+                                    # message saying this value differs for test/beta.
+                                    if key == "metaFields":
+                                        print(key, mach1_name + " and " + mach2_name + " differ")
+                                    # Otherwise print out the field that differs and what the values
+                                    # are for test/beta
+                                    else:
+                                        print_diffs(key, mach2_name, val2, mach1_name, val1)
+                            except KeyError:
+                                print(key, f'present on {mach1_name}, but not on {mach2_name} or vice versa')
+        except KeyError:
+            print(dataset, f'present on {mach1_name}, but not on {mach2_name} or vice versa')
+
 
 # Set up script arguments
 parser = argparse.ArgumentParser(
     formatter_class=argparse.RawDescriptionHelpFormatter,
-    description="Shows diffs between cells-test and cells-beta.")
+    description="Shows diffs between cells-test and cells-beta. By default shows only names ")
 parser.add_argument("-r","--run", action='store_true',
     help='run script to looks for diffs')
-parser.add_argument("-n","--namesonly", action='store_true',
-    help='Show only names of datasets that have diffs')
+parser.add_argument("-v","--verbose", action='store_true',
+    help='Show fields in dataset.json that are different rather than just those datasets that have differences')
+parser.add_argument("-b","--betarr", action='store_true',
+    help='Print diffs between cells-beta and cells')
+parser.add_argument("-s","--stats", action='store_true',
+    help='Print stats about the number of datasets/collections on each machine')
+parser.add_argument("-d","--hidden", action='store_true',
+    help='Print datasets that are hidden on each machine')
 args = parser.parse_args()
 
 def main():
@@ -36,11 +83,50 @@ def main():
         # Hard coded since I don't think you'd ever be able to reasonably compare
         # two arbitrary hosts or collections of datasets?
         # Loading them as dictionaries via json.load
-        ctfh =  open("/usr/local/apache/htdocs-cells/dataset.json", "r")
+        tdir = "/usr/local/apache/htdocs-cells/"
+        ctfh =  open(tdir + "dataset.json", "r")
         # Loading as dictionary via json.load
         cells_test = json.load(ctfh)
-        cbfh =  open("/usr/local/apache/htdocs-cells-beta/dataset.json", "r")
+        # Go Look for datasets hidden on cells-test
+        ct_hide = dict()
+        # Go through all dirs in htdocs-cells
+        for dname in os.listdir(tdir):
+            dpath = tdir + dname + "/dataset.json"
+            # Some dirs in htdocs-cells (e.g. js) aren't dataset dirs
+            # so we need to check paths we're building are to valid files
+            # or else python errors outs
+            if os.path.exists(dpath):
+                with open(dpath) as dfh:
+                    # Hidden datasets have "hide" somewhere in their dataset.json file
+                    if 'hide' in dfh.read():
+                        # Save the names/paths of these json files to a dictionary
+                        ct_hide[dname] = json.load(open(dpath, "r"))
+
+        bdir = "/usr/local/apache/htdocs-cells-beta/"
+        cbfh =  open(bdir + "dataset.json", "r")
         cells_beta = json.load(cbfh)
+        # Look for datasets hidden on cells-test
+        cb_hide = dict()
+        for dname in os.listdir(bdir):
+            dpath = bdir + dname + "/dataset.json"
+            if os.path.exists(dpath):
+                with open(dpath) as dfh:
+                    if 'hide' in dfh.read():
+                        cb_hide[dname] = json.load(open(dpath, "r"))
+
+        rrdir = "https://cells.ucsc.edu/"
+        rr_hide = dict()
+        # Go through all dirs in htdocs-cells
+        bash_cmd = 'grep -l hide /usr/local/apache/htdocs-cells/*/dataset.json | cut -f6 -d "/"'
+        cmd = ['ssh', 'qateam@hgw0', bash_cmd]
+        p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        cmdout, cmderr = p.communicate()
+        datasets = cmdout.decode().strip().split('\n')
+        for dname in datasets:
+            dpath = rrdir + dname  + "/dataset.json"
+            with urllib.request.urlopen(dpath) as url:
+                djson = json.loads(url.read().decode())
+                rr_hide[dname] = djson
 
         # Make some vars for cells-test info:
         #   - a set of datasets and
@@ -54,10 +140,13 @@ def main():
             # Save dataset names to a set
             ct_datasets.add(dname)
             # Load json for current dataset and save into a dict
-            dpath = "/usr/local/apache/htdocs-cells/" + dname + "/dataset.json"
+            dpath = tdir + dname + "/dataset.json"
             dfh = open(dpath, "r")
             djson = json.load(dfh)
             ct_djson[dname] = djson
+        # combine public/hidden dictionaries
+        ct_djson.update(ct_hide)
+        ct_datasets = ct_datasets.union(set(ct_hide.keys()))
 
         # Make some vars for cells-beta info:
         #   - a set of datasets and
@@ -68,73 +157,127 @@ def main():
         for dataset in cells_beta["datasets"]:
             dname = dataset["name"]
             cb_datasets.add(dname)
-            dpath = "/usr/local/apache/htdocs-cells-beta/" + dname + "/dataset.json"
+            dpath = bdir + dname + "/dataset.json"
             dfh = open(dpath, "r")
             djson = json.load(dfh)
             cb_djson[dname] = djson
+        # combine public/hidden dictionaries
+        cb_djson.update(cb_hide)
+        cb_datasets = cb_datasets.union(set(cb_hide.keys()))
+
+        #from https://stackoverflow.com/questions/12965203/how-to-get-json-from-webpage-into-python-script
+        # download/parse dataset.json on the RR
+        rr = dict()
+        rr_datasets = set()
+        rr_djson = dict()
+        with urllib.request.urlopen("https://cells.ucsc.edu/dataset.json") as url:
+            rr = json.loads(url.read().decode())
+        for dataset in rr["datasets"]:
+            dname = dataset["name"]
+            rr_datasets.add(dname)
+            dpath = "https://cells.ucsc.edu/" + dname  + "/dataset.json"
+            with urllib.request.urlopen(dpath) as url:
+                djson = json.loads(url.read().decode())
+                rr_djson[dname] = djson
+        rr_djson.update(rr_hide)
+        rr_datasets = rr_datasets.union(set(rr_hide.keys()))
+
+        if args.hidden:
+            print("cells-test hidden datasets:")
+            print(sorted(list(ct_hide.keys())),"\n")
+            print("cells-beta hidden datasets:")
+            print(sorted(list(cb_hide.keys())),"\n")
+            print("cells hidden datasets:")
+            print(sorted(list(rr_hide.keys())),"\n")
 
         # Print out only datasets on cells-test only
         dev_only = ct_datasets.difference(cb_datasets)
         print("Datasets on cells-test only:")
-        print(dev_only,"\n")
+        print(sorted(list(dev_only)),"\n")
+
+        beta_only = cb_datasets.difference(rr_datasets)
+        print("Datasets on cells-beta, but not on RR:")
+        print(sorted(list(beta_only)),"\n")
+
 
         # First pass of processing dicts to find collections and add subdatasets to main dict
+        n_cb_collections = 0 # count up number of collections on cells-beta so that we can exclude these from calcs later on
         cb_subdirs = dict()
-        ct_subdirs = dict()
         for dataset in cb_djson:
             # Collections have "datasets" somewhere in their keys
             if "datasets" in cb_djson[dataset].keys():
                 subdirs = cb_djson[dataset]["datasets"]
+                n_cb_collections += 1
                 for subdir in subdirs:
                     subname = subdir["name"]
                     # Save info about beta version of dataset
-                    bpath = "/usr/local/apache/htdocs-cells-beta/" + dname + "/dataset.json"
+                    #bpath = bdir + dname + "/" + subname+ "/dataset.json"
+                    bpath = bdir + subname + "/dataset.json"
                     bfh = open(bpath, "r")
                     bjson = json.load(bfh)
                     cb_subdirs[subname] = bjson
 
+        n_ct_collections = 0
+        ct_subdirs = dict()
+        for dataset in ct_djson:
+            # Collections have "datasets" somewhere in their keys
+            if "datasets" in ct_djson[dataset].keys():
+                subdirs = ct_djson[dataset]["datasets"]
+                n_ct_collections += 1
+                for subdir in subdirs:
                     # Save info about test version of dataset
-                    tpath = "/usr/local/apache/htdocs-cells/" + dname + "/dataset.json"
+                    subname = subdir["name"]
+                    tpath = tdir + subname + "/dataset.json"
                     tfh = open(tpath, "r")
                     tjson = json.load(tfh)
                     ct_subdirs[subname] = tjson
 
+        n_rr_collections = 0
+        rr_subdirs = dict()
+        for dataset in rr_djson:
+            # Collections have "datasets" somewhere in their keys
+            if "datasets" in rr_djson[dataset].keys():
+                subdirs = rr_djson[dataset]["datasets"]
+                n_rr_collections += 1
+                for subdir in subdirs:
+                    # Save info about rr version of dataset
+                    subname = subdir["name"]
+                    rrpath = rrdir + subname + "/dataset.json"
+                    with urllib.request.urlopen(rrpath) as url:
+                        rrjson = json.loads(url.read().decode())
+                        rr_subdirs[subname] = rrjson
+
         # Merge these subdir dicts with the main dicts
         # Syntax from https://stackoverflow.com/questions/38987/how-do-i-merge-two-dictionaries-in-a-single-expression-in-python-taking-union-o
         cb_djson = {**cb_djson, **cb_subdirs}
         ct_djson = {**ct_djson, **ct_subdirs}
+        rr_djson = {**rr_djson, **rr_subdirs}
 
-        # Loop through beta datasets as we want to compare only the datasets currently on 
-        # cells-beta to cells-test to look for diffs
-        for dataset in cb_djson:
-            # Check if the overall dicts are the same for a dataset on test/beta
-            # If not, we'll go through and print out only those values that are different
-            if not operator.eq(ct_djson[dataset], cb_djson[dataset]):
-                # Print dataset name
-                print("\n" + dataset)
+        # Compare cells-test to cells-beta datasets
+        compare_machines(ct_djson, "cells-test", cb_djson, "cells-beta", args.verbose)
+        # This line would do the same for cells and cells-beta, but it seems to show diffs for all datasets?
+        # Not sure what's going on with this, but will need to check with Max
+        if args.betarr:
+            compare_machines(cb_djson, "cells-beta", rr_djson, "cells", args.verbose)
+
+        # Print stats about number of datasets on each machine if arg set
+        if args.stats:
+            print("Num of top-level datasets/collections")
+            print("\tcells-test:", len(ct_datasets))
+            print("\tcells-beta:", len(cb_datasets))
+            print("\tcells:", len(rr_datasets))
+            # Calculation excludes top-level collection entries in calculation
+            # they're not really 'datasets', rather what's in a collection is the dataset
+            print("Num of datasets (including those in collections):")
+            print("\tcells-test:", len(ct_djson.keys())-n_ct_collections)
+            print("\tcells-beta:", len(cb_djson.keys())-n_cb_collections)
+            print("\tcells:", len(rr_djson.keys())-n_rr_collections)
+            # Calculation includes top-level collection entries
+            #print("Num of datasets (including those in collections):")
+            #print("\tcells-test:", len(ct_djson.keys()))
+            #print("\tcells-beta:", len(cb_djson.keys()))
+            #print("\tcells:", len(rr_djson.keys()))
 
-                # Script has option to print only names of datasets that have diffs
-                # This whole chunk gets skipped if that's the case
-                if not args.namesonly:
-                    for key in cb_djson[dataset].keys():
-                        # Check if current value is a dict and if so, process that a certain way
-                        if isinstance(cb_djson[dataset][key], dict):
-                            process_dict(cb_djson[dataset][key], ct_djson[dataset][key])
-                        # Otherwise check to see if values for test/beta are different
-                        # and print only the diffs
-                        else: 
-                            beta_val = cb_djson[dataset][key]
-                            test_val = ct_djson[dataset][key]
-                            if beta_val != test_val:
-                                # The value for this keys if often large and bloats the results
-                                # making it difficult to actually see the diffs. Only print a
-                                # message saying this value differs for test/beta.
-                                if key == "metaFields":
-                                    print(key, "beta and test differ")
-                                # Otherwise print out the field that differs and what the values
-                                # are for test/beta
-                                else:
-                                    print_diffs(key, beta_val, test_val)
     else:
         print("Script looks for differences between cells-test and cells-beta.\n\nRun script with -r/--run to look for diffs.")
         exit(1)
diff --git a/ucsc/generateColorHtml b/ucsc/generateColorHtml
new file mode 100755
index 00000000..967bdfe9
--- /dev/null
+++ b/ucsc/generateColorHtml
@@ -0,0 +1,32 @@
+#!/usr/bin/env python3
+
+import argparse, pathlib
+
+parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    description="Creates blocks of html code for colors on track description pages.")
+parser.add_argument('colorFile', type=str, help='Two column file (csv or tsv), can be same as \
+    used in cell browser. Column 1 is metadata value (e.g. cortex-dev). Column 2 is a hex color.')
+
+args = parser.parse_args()
+
+#colors = dict()
+
+# From https://stackoverflow.com/questions/541390/extracting-extension-from-filename-in-python
+fileType = pathlib.Path(args.colorFile).suffix
+
+# It's assumed that the colors file contains hexcodes, though maybe we should detect that
+# and convert via webcolors if needed
+dfh = open(args.colorFile, 'r')
+
+print("<p>\n<ul>")
+for line in dfh:
+    if fileType == ".csv":
+        splitLine = line.strip().split(",", 1)
+    if fileType == ".tsv" or fileType == '.tab':
+        splitLine = line.strip().split("\t", 1)
+    name = splitLine[0]
+    color = splitLine[1]
+    print("<li><span style='border: 2px inset #000000;\n",
+    "background-color:" + color + ";'>&nbsp;&nbsp;</span> - " + name)
+print("</ul>")
diff --git a/ucsc/getTagVals b/ucsc/getTagVals
new file mode 100755
index 00000000..fe8e511e
--- /dev/null
+++ b/ucsc/getTagVals
@@ -0,0 +1,90 @@
+#! /usr/bin/env python3
+
+import sys,argparse,re
+from collections import OrderedDict
+from os.path import join, basename, dirname
+from pathlib import Path
+
+# These two functions copied and modified from cellbrowser.py
+def readLines(lines, fname):
+    " recursively read lines from fname, understands lines like #include 'filename.conf' "
+    for line in open(fname):
+        line = line.rstrip("\r\n")
+        if line.startswith("#include"):
+            newFname = splitOnce(line, " ")[1]
+            newFname = newFname.strip('"').strip("'")
+            lines.extend(readLines(lines, newFname))
+        else:
+            lines.append(line)
+    return lines
+
+def loadConfig(fname):
+    """ parse python in fname and return variables as dictionary.
+    add the directory of fname to the dict as 'inDir'.
+    """
+
+    conf = OrderedDict()
+
+    g = {}
+    g["fileBase"] = basename(fname).split('.')[0]
+    g["dirName"] = basename(dirname(fname))
+
+    lines = readLines([], fname)
+    exec("\n".join(lines), g, conf)
+
+    return conf
+
+# Set up script arguments
+parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    description="Get all values for a list of datasets for specified cellbrowser.conf tag")
+parser.add_argument('datasetFile', type=str, help='file containing list of datasets in first column')
+parser.add_argument('tag', type=str, help='cellbrowser.conf tag you want values for, e.g. body_parts')
+args = parser.parse_args()
+
+# From https://stackoverflow.com/questions/541390/extracting-extension-from-filename-in-python
+dFtype = Path(args.datasetFile).suffix
+
+if dFtype == ".csv":
+    dsep = ","
+elif dFtype == ".tsv":
+    dsep = "\t"
+else:
+    print("datasetFile must have tsv or csv file extension")
+    exit(1)
+
+dfh = open(args.datasetFile)
+
+# If datasets file has info for the tag already
+# get column num so we can replace it
+header = dfh.readline().rstrip().split(dsep)
+try:
+    n = header.index(args.tag)
+# If it's not there, we'll just tack it on to the end
+except ValueError:
+    header.append(args.tag)
+    n = len(header)
+# Want the header in the output file so print that here
+print(*header, sep=dsep)
+
+# Now process rest of datasetFile
+for line in dfh:
+    splitLine = line.rstrip().split(dsep)
+    dname = splitLine[0]
+    confName = join("/hive/data/inside/cells/datasets",dname,"cellbrowser.conf")
+    conf = loadConfig(confName)
+    for tag in header:
+        if tag != "dataset":
+            n = header.index(tag)
+            if tag in conf.keys():
+                tagVal = str(conf[tag])
+                # remove all the extra characters from the line
+                tagVal = re.sub('[\[\]\'",]','',tagVal)
+                # Case where we're tacking the tag onto the end of our file
+                if n == len(header) - 1:
+                    splitLine.append(tagVal)
+                    print(*splitLine, sep=dsep)
+                # Case where we're replacing what's there
+                else:
+                    splitLine[n] = tagVal
+                    print(*splitLine, sep=dsep)
diff --git a/ucsc/h5adMetaInfo b/ucsc/h5adMetaInfo
new file mode 100755
index 00000000..e0426a5e
--- /dev/null
+++ b/ucsc/h5adMetaInfo
@@ -0,0 +1,37 @@
+#! /usr/bin/env python3
+
+import anndata, sys, argparse
+    
+def main():
+    parser = argparse.ArgumentParser(
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        description="Print out info about metadata in an h5ad file")
+    parser.add_argument('h5ad', type=str, help='an h5ad file')
+    parser.add_argument('-f', '--fields', type=int, default=5, \
+      help='number of values to print for a field')
+    args = parser.parse_args()
+
+    ad = anndata.read_h5ad(args.h5ad)
+
+    # Prints out general info about the matrix/dataset
+    print(ad,"\n")
+
+    # Loop goes through each field in obs and prints out at least 5 values
+    for annot in ad.obs_keys():
+        # Get list of unique vals in obs annotation
+        annot_vals = sorted(list(set(ad.obs[annot])))
+        val_count = len(annot_vals)
+        # Can't color on annots with >500 vals, so no need to display them here
+        if val_count < 500:
+            print(annot, "-", val_count, "unique values")
+            for i in range(0,args.fields):
+                # try/except here is because some annots have less than the 
+                # amount we want to print, without this it crashes on these fields
+                try:
+                    print(annot_vals[i])
+                except:
+                    continue
+            print()
+
+if __name__ == "__main__":
+    main()
diff --git a/ucsc/makeCbHub b/ucsc/makeCbHub
new file mode 100755
index 00000000..b58c5c17
--- /dev/null
+++ b/ucsc/makeCbHub
@@ -0,0 +1,183 @@
+#! /usr/bin/env python3
+
+import sys,os,subprocess,webcolors,argparse,pathlib
+
+# Set up script arguments
+parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    # Should probably expand this beyond just bw files in the future?
+    description="Will take a set of bigWig files and turn them into a set of composite tracks,\
+    one per directory. If no subdirs/dataset names provided, then will make a single composite.")
+parser.add_argument('fileDir', type=str, help='name of directory where bw files are')
+parser.add_argument("-d","--datasetList", type=str , help='dataset name or list of datasets. If list,\
+    then dataset names should be enclosed in a set of quotes and space separated, e.g. "H3K27me3 \
+    H3K36me3". These names should match the names of directories in fileDir. Will be used as composite track names.')
+parser.add_argument("-c","--colors", type=str , help='file containing color associations. \
+    Names in file should match those of the file names minus the file extension or match \
+    those provided in -s/--shortLabels.')
+parser.add_argument("-s","--shortLabels", type=str , help="if file names aren't what is desired \
+    for track names, can provided a file containing file names in column 1 and desired track \
+    names in column 2.")
+parser.add_argument("-l","--longLabels", type=str , help="if file names aren't what is desired \
+    for track names, can provided a file containing file names in column 1 and desired track \
+    names in column 2.")
+parser.add_argument("-f","--html", type=str , help="track description html file. Assumes same \
+    one to be used by all.")
+
+args = parser.parse_args()
+
+dsList = list()
+# Handle case where people provide a list of track/dir names
+if args.datasetList:
+    dsList = args.datasetList.strip().split()
+# Otherwise, just assume track names == dir names
+else:
+    for d in os.listdir(args.fileDir):
+        if os.path.isdir(os.path.join(args.fileDir, d)):
+            dsList.append(d)
+# if dir has no subdirs, then just set dsList == input file dir
+if len(dsList) == 0:
+    dsList = [args.fileDir]
+
+colors = dict()
+if args.colors:
+    # get suffix of colors file to know delimeter, i.e. tab-sep vs comma-sep
+    # From https://stackoverflow.com/questions/541390/extracting-extension-from-filename-in-python
+    cFtype = pathlib.Path(args.colors).suffix
+    if cFtype == ".csv":
+        csep = ","
+    elif cFtype == ".tsv":
+        csep = "\t"
+    else:
+        print("colors file must have tsv or csv file extension")
+        exit(1)
+    #cfh = open(args.colors, "r")
+    for line in open(args.colors, "r"):
+        splitLine = line.strip().split(csep)
+        colors[splitLine[0]] = splitLine[1]
+shortLabels = dict()
+if args.shortLabels:
+    sFtype = pathlib.Path(args.shortLabels).suffix
+    if sFtype == ".csv":
+        ssep = ","
+    elif sFtype == ".tsv":
+        ssep = "\t"
+    else:
+        print("shortLabels file must have tsv or csv file extension")
+        exit(1)
+    #sfh = open(args.shortLabels, "r")
+    for line in open(args.shortLabels, "r"):
+        splitLine = line.strip().split(ssep)
+        shortLabels[splitLine[0]] = splitLine[1]
+
+longLabels = dict()
+if args.longLabels:
+    lFtype = pathlib.Path(args.longLabels).suffix
+    if lFtype == ".csv":
+        lsep = ","
+    elif lFtype == ".tsv":
+        lsep = "\t"
+    else:
+        print("longLabels file must have tsv or csv file extension")
+        exit(1)
+    #lfh = open(args.longLabels, "r")
+    for line in open(args.longLabels, "r"):
+        splitLine = line.strip().split(lsep)
+        longLabels[splitLine[0]] = splitLine[1]
+
+
+for ds in dsList:
+    dirName = ds.lower()
+    #bwDir = dirName + "/bigWig/"
+    # ^^ Could possibily determine this somehow
+    # from walking the directories and finding dirs
+    # w/ big* files?
+
+    # This prints everything for the parent stanza
+    print("track",dirName)
+    print("compositeTrack","on")
+    print("shortLabel", ds)
+    print("longLabel", ds)
+    print("visibility","dense")
+    print("autoScale","group")
+    print("type","bigWig") 
+    # ^^how to determine this from file types below?
+    # Instead of just printing as I go, I guess I could
+    # make a dict or something and fill it up and print
+    # the stanzas at the end. 
+    # Or maybe the type here doesn't even matter? 
+    # Should really double check
+    # Use track html if provided
+    if args.html:
+        print("html", args.html)
+    print()
+    if dsList[0] == args.fileDir:
+        fdir = args.fileDir
+    else:
+        fdir = os.path.join(args.fileDir,dirName)
+
+    # This prints out the stanzas for all of the subtracks
+    for f in os.listdir(fdir):
+        # Basically take everything before final suffix to use as track name since we need something unique
+        cluster = f.rsplit(".",1)[0]
+        ftype = pathlib.Path(f).suffix
+        print("  track", dirName + "_" + cluster)
+        print("  parent", dirName, "on")
+        # If someone specified short labels, use those
+        if len(shortLabels) != 0:
+            try:
+                shortLabel = shortLabels[f]
+            except KeyError:
+                shortLabel = cluster
+        # Otherwise just use the file name
+        else:
+            shortLabel = cluster
+        print("  shortLabel", shortLabel)
+        # Again, if someone specifies long labels, use those
+        if len(longLabels) != 0:
+            # need try/except here in cases where not every short label has a corresponding long label
+            # otherwise you get a KeyError and the script just crashes
+            try:
+                longLabel = shortLabel + " - " + longLabels[shortLabel]
+            except KeyError:
+                longLabel = shortLabel
+        # If no long labels specified, just use the short label
+        else:
+            longLabel = shortLabel
+        if dsList[0] == args.fileDir:
+            print("  longLabel", longLabel)
+        else:
+            print("  longLabel", ds + " - " + longLabel)
+        fpath = os.path.join(fdir,f)
+        # Do a bunch of things specific to bigWig files
+        if ftype in {".bw",".bigWig",".bigwig"}:
+            # Calculate min/max via bigWigInfo since that's required for the type line
+            cmd = ["bigWigInfo", fpath]
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            cmdout, cmderr = p.communicate()
+            results = cmdout.decode("ASCII").split("\n")
+            resultsDict = dict()
+            for item in results:
+                splitItem = item.replace(" ","").split(":")
+                if len(splitItem) > 1:
+                    resultsDict[splitItem[0]] = splitItem[1]
+            print("  type","bigWig",resultsDict["min"],resultsDict["max"])
+            # autoScale group: http://genome.ucsc.edu/goldenPath/help/trackDb/trackDbHub.html#autoScale
+            print("  autoScale","group")
+        # An elif where we could do stuff specific to bigBeds
+        elif ftype in {".bb", ".bigBed"}:
+            print("I don't do anything yet")
+        # If someone specifies colors, use those
+        if len(colors) != 0:
+            # Assumes that input colors are in hex format, but GB only takes rgb values
+            # Should probably have it auto-detect that here and not convert if not needed
+            if shortLabel in colors.keys():
+                rgbTuple = webcolors.hex_to_rgb(colors[shortLabel])
+                rgbStr = str(rgbTuple.red) + "," + str(rgbTuple.green) + "," + str(rgbTuple.blue)
+                print("  color", rgbStr)
+        print("  bigDataUrl",fpath)
+        print("  visibility","dense")
+        # Need this empty print here so that a space is printed betwen track stanzas
+        print()
+
+       
diff --git a/ucsc/updateNewsSec b/ucsc/updateNewsSec
new file mode 100755
index 00000000..731ccc9a
--- /dev/null
+++ b/ucsc/updateNewsSec
@@ -0,0 +1,270 @@
+#! /usr/bin/env python3
+
+import json, sys, operator, argparse, os, urllib.request, subprocess, time, glob
+from datetime import datetime, timedelta, date
+
+parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    description="Update 'News' section in desc.conf. Also updates sitemap and list of datasets currently on the RR.")
+parser.add_argument("-r","--run", action='store_true',
+    help='run script to update news, rr datasets list, and sitemap.')
+args = parser.parse_args()
+
+def buildSitemapSet():
+    """Builds a set of all URLs currently in the sitemap."""
+
+    sitemapPath = "/hive/data/inside/cells/sitemap.cells.txt"
+    sitemapSet = set()
+
+    # If the sitemap file exists, then open it and add each line to a new set
+    if os.path.exists(sitemapPath):
+        sitemap = open(sitemapPath, "r")
+        for line in sitemap:
+            sitemapSet.add(line)
+        sitemap.close()
+    # Otherwise, just make an empty file?
+    # Maybe I should try/except here instead?
+    else:
+        sitemap = open(sitemapPath, "w")
+        sitemap.close()
+
+    return sitemapSet
+
+def addSiteToSitemap(entry, sitemapSet):
+    """Writes an entry out to the sitemap if it's not already there."""
+
+    # Open sitemap file
+    with open("/hive/data/inside/cells/sitemap.cells.txt", "a") as sitemap:
+        urlline = "https://" + entry + ".cells.ucsc.edu\n"
+        if urlline not in sitemapSet:
+            sitemap.write(urlline)
+
+def makeDate(aDate):
+    """Will turn a string date separated by "-" into a date object."""
+
+    # to make a date object, we need at least the year, month, and day
+    splitDate = aDate.split("-")
+    year = int(splitDate[0])
+    month = int(splitDate[1])
+    day = int(splitDate[2])
+    dateObj = date(year, month, day)
+
+    return dateObj
+
+def mondayBefore(aDate):
+    """Returns the date of the monday before a given date."""
+
+    # Use makeDate function to make input date into a datetime object
+    dateObj = makeDate(aDate)
+    # From https://www.programiz.com/python-programming/datetime
+    # Basically allows us to get the date for the monday in which
+    # a given date falls
+    monday = dateObj - timedelta(days=dateObj.weekday())
+
+    return monday
+
+def processDataset(dataset, rootDir):
+    """Processes a single dataset and returns the shortname and a list of info about that dataset."""
+
+    name = dataset["name"]
+    # 'firstBuildTime was added by Max in early 2022, so not all datasets have it
+    # Need to be resistant to it not being there
+    try:
+        lastmtime = dataset["firstBuildTime"]
+    except KeyError:
+        lastmtime = time.strftime('%Y-%m-%d', time.localtime(os.path.getmtime(os.path.join(rootDir, name, "exprMatrix.bin"))))
+    monday = mondayBefore(lastmtime)
+
+    # Collect info for dataset to return at end of function
+    short = dataset["shortLabel"]
+    count = str(dataset["sampleCount"])
+    dList = [short, count, monday]
+
+    return name, dList
+
+def processCollection(collection, rootDir):
+    """Processes a collection and returns a list of info about that collection.
+       If it runs into a nested collection, will also process that."""
+
+    collInfo = list()
+
+    name = collection["name"]
+    cjson = json.load(open(os.path.join(rootDir, name, "dataset.json"), "r"))
+    subdirs = cjson["datasets"]
+
+    for sub in subdirs:
+        if 'isCollection' in sub.keys():
+            # If we run into another collection, process it in the same way
+            subCollInfo = processCollection(sub, rootDir)
+            # Need to do this appending here otherwise subcollection info disappears
+            collInfo = collInfo + subCollInfo
+
+        else:
+            # Otherwise, process each dataset in a collection as a normal dataset
+            subInfo = processDataset(sub, rootDir)
+            subDict = {subInfo[0]:subInfo[1]}
+            collInfo.append(subDict)
+
+    return collInfo
+
+def parseBetaDatasets():
+    """Parses dataset.json for cells-beta to get a current list of datasets."""
+
+    # Builds dictionary of datasets currently on the beta, assuming it's a proxy of the RR
+    betaInfo = dict() # a smaller dict that contains only name/shortLabel/count/release date
+    bdir = "/usr/local/apache/htdocs-cells-beta/"
+    cellsBeta =  json.load(open(os.path.join(bdir, "dataset.json"), "r"))
+    for dataset in cellsBeta["datasets"]:
+        if "isCollection" in dataset.keys():
+            collectionInfo = processCollection(dataset, bdir)
+            cname = dataset["name"]
+            cshort = dataset["shortLabel"]
+
+            oldDate = date.today()
+            collCellCount = 0
+            # After we process the collection, we want to add the info for that
+            # to the betaInfo dictionary
+            for d in collectionInfo:
+                dname = list(d.keys())[0]
+                info = d[dname]
+                betaInfo[dname] = info
+
+                collCellCount += int(info[1])
+                day = info[2]
+                # Do some comparison to see if data for current dataset in collection
+                # is older than the last
+                if day < oldDate:
+                    oldDate = day
+            # Also add an entry to betaInfo that covers the collection as a whole
+            # Cell count for this one is the sum of the cell counts for all subdatasets
+            # Date is that for the oldest dataset in the collection
+            betaInfo[cname] = [cshort, str(collCellCount), oldDate]
+        else:
+            dname, bList = processDataset(dataset, bdir)
+            betaInfo[dname] = bList
+
+    return betaInfo
+
+def writeNewsHtml(toPrint, dateDir):
+    """Takes a list of datasets and writes out an html file per day that lists
+       all datasets released that day."""
+
+    for day in toPrint:
+        dateOut = dateDir + str(day) + ".html"
+        if os.path.exists(dateOut):
+            htmlOut = open(dateOut, "a")
+        else:
+            htmlOut = open(dateOut, "w")
+
+        # Do some work to get the date into something we can easily grab pieces of
+        betterDate = time.strftime('%d-%b-%Y', day.timetuple())
+        splitDay = betterDate.split("-")
+        # Separate vars for month/day/year
+        month=splitDay[1]
+        dayNum=splitDay[0]
+        year=splitDay[2]
+
+        # Write bits out to the news file for the specific day
+        htmlOut.write("<p><b>" + month + " " + dayNum + ", " + year + "</b></p>\n")
+        htmlOut.write("<p>New datasets:</p>\n<ul>\n")
+        for line in toPrint[day]:
+            htmlOut.write(line)
+        htmlOut.write("</ul>\n")
+        htmlOut.close()
+    # From https://stackoverflow.com/questions/3207219/how-do-i-list-all-files-of-a-directory
+    newsFiles = sorted(glob.glob("/hive/data/inside/cells/news/perDate/*.html"), reverse=True)
+    # Basically, we're gathering up all of the individual news files to combine them into one
+    filenames = ['/hive/data/inside/cells/news/basic.html'] + newsFiles
+    with open('/hive/data/inside/cells/news/combined.html','w') as outfile:
+        for fname in filenames:
+            with open(fname) as infile:
+                outfile.write(infile.read())
+
+def main():
+    if args.run == True:
+        # From https://stackoverflow.com/questions/19216334/python-give-start-and-end-of-week-data-from-a-given-date
+        # and https://www.programiz.com/python-programming/datetime/current-datetime
+        # Get date for Monday, so that all datasets added in the last week show up under the same date
+        start = mondayBefore(date.today().strftime('%Y-%m-%d'))
+
+        # File should contain RR datasets
+        # First run of this script will generate this file,
+        # move it out of the way to regenerate, though this means that
+        # everything will be noted as being released on the same day
+        rrDatasetsPath = "/hive/data/inside/cells/rr.datasets.txt"
+        dateDir = "/hive/data/inside/cells/news/perDate/"
+
+        sitemapSet = buildSitemapSet()
+
+        # This should only happen if this is the first time the script is run
+        # or if previous version is moved/deleted
+        if not os.path.exists(rrDatasetsPath):
+            betaInfo = parseBetaDatasets()
+            toPrint = dict()
+            for entry in betaInfo.keys():
+                label = betaInfo[entry][0]
+                count = betaInfo[entry][1]
+                day = betaInfo[entry][2]
+                line =  str(day) + "\t" + entry + "\t" + label + "\t" + count + "\n"
+                with open(rrDatasetsPath, "a") as rrDatasets:
+                    rrDatasets.write(line)
+                # Check if '/' in shortname, if so means it's a dataset in a collection
+                # and we're not outputting it to the sitemap or to the announcements
+                if "/" not in entry:
+                    addSiteToSitemap(entry, sitemapSet)
+                    outLine = "  <li><a href='?ds=" + entry + "' target='_blank'>" + label + "</a>\n"
+                    if day not in toPrint.keys():
+                        toPrint[day] = [outLine]
+                    else:
+                        toPrint[day].append(outLine)
+
+            writeNewsHtml(toPrint, dateDir)
+
+        else: # This is the main part of the function that prints out the html for a news update
+            betaInfo = parseBetaDatasets()
+
+            # Parse the old rr.datasets.txt file so we know what's already out there
+            oldNames = set()
+            oldDatasets = open(rrDatasetsPath,"r")
+            for line in oldDatasets:
+                splitLine = line.strip().split("\t")
+                name = splitLine[1]
+                oldDate = makeDate(splitLine[0])
+                # Remove entry from betaInfo dict if it existed in rrDatasetsFile
+                if name in betaInfo.keys():
+                    del betaInfo[name]
+                oldNames.add(name)
+            oldDatasets.close()
+
+            sitemapSet = buildSitemapSet()
+
+            if len(betaInfo) > 0:
+                allDatasets = open(rrDatasetsPath,"a")
+                # Go through and determine which datasets need to be added to the news announcements
+                toPrint = dict()
+                for entry in betaInfo:
+                    label = betaInfo[entry][0]
+                    count = betaInfo[entry][1]
+                    day = betaInfo[entry][2]
+                    line = str(day) + "\t" + entry + "\t" + label + "\t" + count + "\n"
+                    if entry not in oldNames:
+                        allDatasets.write(line)
+                    # Check if '/' in shortname, if so means it's a dataset in a collection
+                    # and we're not outputting it to the sitemap or to the announcements
+                    if "/" not in entry:
+                        addSiteToSitemap(entry, sitemapSet)
+                        outLine = "  <li><a href='?ds=" + entry + "' target='_blank'>" + label + "</a>\n"
+                        # If doesn't already in exist in toPrint, add it
+                        if day not in toPrint.keys():
+                            toPrint[day] = [outLine]
+                        else:
+                            toPrint[day].append(outLine)
+
+                # Print out HTML for new datasets to be put into /hive/data/inside/cells/datasets/desc.conf
+                writeNewsHtml(toPrint, dateDir)
+    else:
+        parser.print_help(sys.stderr)
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/test/runTests.sh b/test/runTests.sh
index 8808fac8..29b63e58 100755
--- a/test/runTests.sh
+++ b/test/runTests.sh
@@ -6,6 +6,7 @@ if [ ! -f pbmc_small/exprMatrix.tsv.gz ]; then
 fi
 ../src/cbScanpy -e pbmc_small/exprMatrix.tsv.gz -o scanpyout -n pbmcSmall 
 #../src/cbImportScanpy -i scanpyout/anndata.h5ad -o importScanpyOut
-../src/cbSeurat -e exprMatrix.tsv.gz -o seuratOut -n pbmcSmallSeurat
+../src/cbImportScanpy -i scanpyout/anndata.h5ad -o importScanpyOut -f mtx
+../src/cbSeurat -e pbmc_small/exprMatrix.tsv.gz -o seuratOut -n pbmcSmallSeurat
 #../src/cbImportSeurat -i seurat2.rds -o cbImportSeurat
 #cd 5kPbmcMtx/ && cbTool mtx2tsv matrix.mtx.gz barcodes.tsv.gz features.tsv.gz exprMatrix.tsv.gz && cd ..