Package: ldmat
Efficient Storage and Querying of Linkage Disequilibrium MatricesSetupInstall the package with:pip install ldmatDownloading Example FilesTHIS IS A TEMPORARY WORKAROUND UNTIL A MORE PERMANENT STORAGE SOLUTION IS FOUND.Run the following commands to create a directory calledexampleswith all of the example and test files in it.mkdir ldmat_tmp
cd ldmat_tmp
git init
git config core.sparseCheckout true
git remote add origin https://github.com/G2Lab/ldmat.git
echo "examples/" > .git/info/sparse-checkout
git fetch --depth 1 origin
git pull origin main
cd ..
mv ldmat_tmp/examples .
rm -rf ldmat_tmpGetting StartedTo see all the commands available, runldmat.For any specific command, you can get more information with the--helpflag, like:ldmat submatrix --help.The package includes someexample files.The unprocessed LD matrices are too large to include here, but there is a sample
processed matrix,chr21_partial.h5, which includes LD values for chromosome 21
from positions 13000001 to 22000001 (although there is no data between positions 13000001 and 14000001).
This file contains all LD values greater than 0.1, rounded to 2 decimals.Sample QueryFor a sample query, we can extract the square submatrix of positions 14300001 through 14400001 with the following command:ldmat submatrix examples/chr21_partial.h5 --row-start 14300001 --row-end 14400001This will simply print the results as a Pandas DataFrame, so you'll probably want
to save the results by adding an output file, like:ldmat submatrix ... -o YOUR_OUTPUT.csvSample CompressionTo try compressing a file, we've included a single LD matrix with mock data, generated from a very approximate
Gaussian fit of the values in an actual LD matrix. In order to convert it to the compressed format, run:ldmat convert-chromosome "examples/chr0_*.npz" YOUR_OUTPUT.h5 -c 0 -d 2 -m .1This will find all files matching "examples/chr0_*.npz" (which is just the single provided file) and compress them,
dropping any values less than 0.1, and only keeping 2 decimals for everything. The output file can then be queried as
described above.Public MethodsYou may wish to select a submatrix without having to read to or write from the filesystem, for example if you are
writing a script which needs to select a submatrix on the fly. For this reason, several ldmat methods have been created
specifically for public use. You can import all of these methods withimport ldmat.select_submatrix_by_range(
    ld_h5: h5py.File,
    row_start: int,
    row_end: int,
    col_start: int,
    col_end: int,
    stream: bool = None,
) -> Union[pd.DataFrame, str]The inputs to this method are:The LD file, opened with h5py, like:import h5py; ld_h5 = h5py.File(ld_file, "r").The start row of the range.The end row of the range.The start column of the range.The end column of the range.Whether to stream the results to a CSV. If not specified, this is decided automatically based on the expected matrix size.select_submatrix_by_list(
    ld_h5: h5py.File,
    row_list: List[int],
    col_list: List[int],
    stream: bool = None,
) -> Union[pd.DataFrame, str]The inputs to this method are:The LD file, opened with h5py, like:import h5py; ld_h5 = h5py.File(ld_file, "r").The list of rows to include.The list of columns to include.Whether to stream the results to a CSV. If not specified, this is decided automatically based on the expected matrix size.select_submatrix_by_maf(
    ld_h5: h5py.File,
    lower_bound: float,
    upper_bound: float,
    stream: bool = None,
) -> Union[pd.DataFrame, str]The inputs to this method are:The LD file, opened with h5py, like:import h5py; ld_h5 = h5py.File(ld_file, "r").The smallest MAF to include.The largest MAF to include.Whether to stream the results to a CSV. If not specified, this is decided automatically based on the expected matrix size.Typically, the results of each of these methods will be a pandas DataFrame. However, if the data is streamed, the result
will be a string, the path to the CSV result.Input LoadersThe default input format to this tool is the files foundhere.
However, CSVs and TSVs are also supported natively, if they are in the same format as the output of thesubmatrixcommands.
In this case, use the--loaderoption to specify the format, like:ldmat convert-chromosome "examples/chr0_*.csv" YOUR_OUTPUT.h5 --loader csvCustom LoadersIf your data is in a different format, you may wish to write a custom loader.
This is easy, and simply requires implementing a few methods in a new subclass ofLoader.The two methods that must be implemented areload_as_sparse_matrixandload_metadata.
If you have a custom MAF format, implementload_mafas well.Finally, make sure to give your loader aFRIENDLY_NAMEso it can be referenced by the CLI.If you think your input format is generic enough, or otherwise likely to be reused, feel free to submit
a pull request to add your loader to the package.