feat: add orthrus logo

README.md

@@ -7,7 +7,12 @@ metrics:
 - pearsonr
 ---
 
+
 ## Model Overview
+
+<div style="display: flex; align-items: flex-start;">
+<div style="flex: 1;">
+
 Orthrus is a mature RNA model for RNA property prediction. It uses a mamba encoder backbone, a variant of state-space models specifically designed for long-sequence data, such as RNA. 
 
 Two versions of Orthrus are available:
@@ -22,6 +27,12 @@ The mamba architecture is an extension of the S4 (structured state-space) model
 - _Variable Context Filtering:_ RNA sequences often contain functionally relevant motifs separated by variable spacing. The mamba model is capable of selectively focusing on these important elements
 - _Selective Context Compression:_ Genomic sequences often have uneven information density, with critical regulatory elements scattered across regions of varying importance. The mamba model selectively compresses less informative regions while preserving the context of key functional areas
 
+</div>
+<div style="flex: 1; margin-left: 20px;">
+<img src="images/orthrus_cleaned_up.png" alt="Orthrus Model Overview" style="max-width: 34%; height: auto;">
+</div>
+</div>
+
 ## Using Orthrus
 
 Orthrus was trained on full RNA sequences, making its usage different from models like DNABERT or Enformer, which focus on arbitrary DNA segments. 
@@ -29,6 +40,39 @@ Orthrus was instead trained on full mature RNA sequences so if you pass an incom
 
 To generate embeddings using Orthrus for spliced mature RNA sequences, follow the steps below:
 
+### Create and Set Up the Environment
+
+We recommend using Mamba (a faster alternative to Conda) for environment creation, but Conda works as well. Follow these steps to set up your environment:
+
+1. Install Mamba or Conda if you haven't already.
+
+2. Create and activate the environment:
+```bash
+# Create the environment from the provided YAML file
+mamba env create -f env.yml
+# Activate the environment
+conda activate orthrus
+```
+
+3. Install additional dependencies:
+```bash
+# Install necessary dependencies for the Mamba model (not the environment manager)
+pip install causal_conv1d==1.1.1
+pip install mamba-ssm==1.2.0.post1 --no-cache-dir
+```
+
+4. Set up GenomeKit:
+```bash
+# Download the GenomeKit setup script
+wget -O starter_build.sh https://raw.githubusercontent.com/deepgenomics/GenomeKit/main/starter/build.sh
+# Make the script executable
+chmod +x starter_build.sh
+# Run the script to download genomes and annotations
+./starter_build.sh
+```
+
+Now you're ready to use Orthrus for generating embeddings!
+
 ### Generating Embeddings
 
 #### 4-Track Model

env.yml

@@ -0,0 +1,18 @@
+name: orthrus
+channels:
+- pytorch
+- conda-forge
+- defaults
+- nvidia
+dependencies:
+  - python=3.10
+  - numpy
+  - pandas
+  - pip
+  - jupyter
+  - notebook
+  - tqdm
+  - pytorch=2.2
+  - pytorch-cuda
+  - cuda-toolkit=12.1
+  - genomekit>=6.0.0

notebooks/colab_prototye.ipynb

@@ -0,0 +1,619 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## GenomeKit foundations\n",
+    "\n",
+    "We're going to be using an awesome library called GenomeKit to extract DNA sequences and build 4/6 track representations of mRNA transcripts, which will be used as input for Orthrus. GenomeKit makes it easy to work with genomic data, such as sequences and annotations, by providing tools to access and manipulate reference genomes and variants efficiently. It's built by the awesome folks at Deep Genomics\n",
+    "\n",
+    "For more details, you can refer to the [GenomeKit documentation](https://deepgenomics.github.io/GenomeKit/api.html).\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import genome_kit as gk\n",
+    "from genome_kit import Genome, Interval\n",
+    "import numpy as np"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'CTCTTATGCTCGGGTGATCC'"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "genome = Genome(\"gencode.v29\")\n",
+    "interval = Interval(\"chr7\", \"+\", 117120016, 117120036, genome)\n",
+    "genome.dna(interval)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "5447\n"
+     ]
+    }
+   ],
+   "source": [
+    "genes = [x for x in genome.genes if x.name == 'PKP1']\n",
+    "transcripts = genes[0].transcripts\n",
+    "t = [t for t in transcripts if t.id == 'ENST00000263946.7'][0]\n",
+    "print(sum([len(x) for x in t.exons]))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def find_transcript(genome, transcript_id):\n",
+    "    \"\"\"Find a transcript in a genome by transcript ID.\n",
+    "    \n",
+    "    Args:\n",
+    "        genome (object): The genome object containing a list of transcripts.\n",
+    "        transcript_id (str): The ID of the transcript to find.\n",
+    "        \n",
+    "    Returns:\n",
+    "        object: The transcript object, if found.\n",
+    "        \n",
+    "    Raises:\n",
+    "        ValueError: If no transcript with the given ID is found.\n",
+    "    \n",
+    "    Example:\n",
+    "        >>> # Create sample transcripts and a genome\n",
+    "        >>> transcript1 = 'ENST00000263946'\n",
+    "        >>> genome = Genome(\"gencode.v29\")\n",
+    "        >>> result = find_transcript(genome, 'ENST00000335137')\n",
+    "        >>> print(result.id)\n",
+    "        <Transcript ENST00000263946.7 of PKP1>\n",
+    "        >>> # If transcript ID is not found\n",
+    "        >>> find_transcript(genome, 'ENST00000000000')\n",
+    "        ValueError: Transcript with ID ENST00000000000 not found.\n",
+    "    \"\"\"\n",
+    "    transcripts = [x for x in genome.transcripts if x.id.split('.')[0] == transcript_id]\n",
+    "    if not transcripts:\n",
+    "        raise ValueError(f\"Transcript with ID {transcript_id} not found.\")\n",
+    "    \n",
+    "    return transcripts[0]\n",
+    "\n",
+    "def find_transcript_by_gene_name(genome, gene_name):\n",
+    "    \"\"\"Find all transcripts in a genome by gene name.\n",
+    "    \n",
+    "    Args:\n",
+    "        genome (object): The genome object containing a list of transcripts.\n",
+    "        gene_name (str): The name of the gene whose transcripts are to be found.\n",
+    "        \n",
+    "    Returns:\n",
+    "        list: A list of transcript objects corresponding to the given gene name.\n",
+    "        \n",
+    "    Raises:\n",
+    "        ValueError: If no transcripts for the given gene name are found.\n",
+    "    \n",
+    "    Example:\n",
+    "        >>> # Find transcripts by gene name\n",
+    "        >>> transcripts = find_transcript_by_gene_name(genome, 'PKP1')\n",
+    "        >>> print(transcripts)\n",
+    "        [<Transcript ENST00000367324.7 of PKP1>,\n",
+    "        <Transcript ENST00000263946.7 of PKP1>,\n",
+    "        <Transcript ENST00000352845.3 of PKP1>,\n",
+    "        <Transcript ENST00000475988.1 of PKP1>,\n",
+    "        <Transcript ENST00000477817.1 of PKP1>]        \n",
+    "        >>> # If gene name is not found\n",
+    "        >>> find_transcript_by_gene_name(genome, 'XYZ')\n",
+    "        ValueError: No transcripts found for gene name XYZ.\n",
+    "    \"\"\"\n",
+    "    genes = [x for x in genome.genes if x.name == gene_name]\n",
+    "    if not genes:\n",
+    "        raise ValueError(f\"No genes found for gene name {gene_name}.\")\n",
+    "    if len(genes) > 1:\n",
+    "        print(f\"Warning: More than one gene found for gene name {gene_name}.\")\n",
+    "        print('Concatenating transcripts from all genes.')\n",
+    "        \n",
+    "    transcripts = []\n",
+    "    for gene in genes:\n",
+    "        transcripts += gene.transcripts\n",
+    "    return transcripts\n",
+    "\n",
+    "def create_cds_track(t):\n",
+    "    \"\"\"Create a track of the coding sequence of a transcript.\n",
+    "    Use the exons of the transcript to create a track where the first position of the codon is one.\n",
+    "    \n",
+    "    Args:\n",
+    "        t (gk.Transcript): The transcript object.\n",
+    "    \"\"\"\n",
+    "    cds_intervals = t.cdss\n",
+    "    utr3_intervals = t.utr3s\n",
+    "    utr5_intervals = t.utr5s\n",
+    "    \n",
+    "    len_utr3 = sum([len(x) for x in utr3_intervals])\n",
+    "    len_utr5 = sum([len(x) for x in utr5_intervals])\n",
+    "    len_cds = sum([len(x) for x in cds_intervals])\n",
+    "    \n",
+    "    # create a track where first position of the codon is one\n",
+    "    cds_track = np.zeros(len_cds, dtype=int)\n",
+    "    # set every third position to 1\n",
+    "    cds_track[0::3] = 1\n",
+    "    # concat with zeros of utr3 and utr5\n",
+    "    cds_track = np.concatenate([np.zeros(len_utr5, dtype=int), cds_track, np.zeros(len_utr3, dtype=int)])\n",
+    "    return cds_track\n",
+    "\n",
+    "def create_splice_track(t):\n",
+    "    \"\"\"Create a track of the splice sites of a transcript.\n",
+    "    The track is a 1D array where the positions of the splice sites are 1.\n",
+    "\n",
+    "    Args:\n",
+    "        t (gk.Transcript): The transcript object.\n",
+    "    \"\"\"\n",
+    "    len_utr3 = sum([len(x) for x in t.utr3s])\n",
+    "    len_utr5 = sum([len(x) for x in t.utr5s])\n",
+    "    len_cds = sum([len(x) for x in t.cdss])\n",
+    "    \n",
+    "    len_mrna = len_utr3 + len_utr5 + len_cds\n",
+    "    splicing_track = np.zeros(len_mrna, dtype=int)\n",
+    "    cumulative_len = 0\n",
+    "    for exon in t.exons:\n",
+    "        cumulative_len += len(exon)\n",
+    "        splicing_track[cumulative_len - 1:cumulative_len] = 1\n",
+    "        \n",
+    "    return splicing_track\n",
+    "\n",
+    "# convert to one hot\n",
+    "def seq_to_oh(seq):\n",
+    "    oh = np.zeros((len(seq), 4), dtype=int)\n",
+    "    for i, base in enumerate(seq):\n",
+    "        if base == 'A':\n",
+    "            oh[i, 0] = 1\n",
+    "        elif base == 'C':\n",
+    "            oh[i, 1] = 1\n",
+    "        elif base == 'G':\n",
+    "            oh[i, 2] = 1\n",
+    "        elif base == 'T':\n",
+    "            oh[i, 3] = 1\n",
+    "    return oh\n",
+    "\n",
+    "def create_one_hot_encoding(t):\n",
+    "    \"\"\"Create a track of the sequence of a transcript.\n",
+    "    The track is a 2D array where the rows are the positions\n",
+    "    and the columns are the one-hot encoding of the bases.\n",
+    "\n",
+    "    Args\n",
+    "        t (gk.Transcript): The transcript object.\n",
+    "    \"\"\"\n",
+    "    seq = \"\".join([genome.dna(exon) for exon in t.exons])\n",
+    "    oh = seq_to_oh(seq)\n",
+    "    return oh\n",
+    "\n",
+    "def create_six_track_encoding(t, channels_last=False):\n",
+    "    \"\"\"Create a track of the sequence of a transcript.\n",
+    "    The track is a 2D array where the rows are the positions\n",
+    "    and the columns are the one-hot encoding of the bases.\n",
+    "    Concatenate the one-hot encoding with the cds track and the splice track.\n",
+    "\n",
+    "    Args\n",
+    "        t (gk.Transcript): The transcript object.\n",
+    "    \"\"\"\n",
+    "    oh = create_one_hot_encoding(t)\n",
+    "    cds_track = create_cds_track(t)\n",
+    "    splice_track = create_splice_track(t)\n",
+    "    six_track = np.concatenate([oh, cds_track[:, None], splice_track[:, None]], axis=1)\n",
+    "    if not channels_last:\n",
+    "        six_track = six_track.T\n",
+    "    return six_track"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import math\n",
+    "from functools import partial\n",
+    "import os\n",
+    "import json\n",
+    "import torch\n",
+    "import torch.nn as nn\n",
+    "\n",
+    "from mamba_ssm.modules.mamba_simple import Mamba, Block\n",
+    "from huggingface_hub import PyTorchModelHubMixin\n",
+    "\n",
+    "def create_block(\n",
+    "    d_model,\n",
+    "    ssm_cfg=None,\n",
+    "    norm_epsilon=1e-5,\n",
+    "    residual_in_fp32=False,\n",
+    "    fused_add_norm=False,\n",
+    "    layer_idx=None,\n",
+    "    device=None,\n",
+    "    dtype=None,\n",
+    "):\n",
+    "    if ssm_cfg is None:\n",
+    "        ssm_cfg = {}\n",
+    "    factory_kwargs = {\"device\": device, \"dtype\": dtype}\n",
+    "    mix_cls = partial(Mamba, layer_idx=layer_idx, **ssm_cfg, **factory_kwargs)\n",
+    "    norm_cls = partial(nn.LayerNorm, eps=norm_epsilon, **factory_kwargs)\n",
+    "    block = Block(\n",
+    "        d_model,\n",
+    "        mix_cls,\n",
+    "        norm_cls=norm_cls,\n",
+    "        fused_add_norm=fused_add_norm,\n",
+    "        residual_in_fp32=residual_in_fp32,\n",
+    "    )\n",
+    "    block.layer_idx = layer_idx\n",
+    "    return block\n",
+    "\n",
+    "\n",
+    "class MixerModel(\n",
+    "    nn.Module,\n",
+    "    PyTorchModelHubMixin,\n",
+    "):\n",
+    "\n",
+    "    def __init__(\n",
+    "        self,\n",
+    "        d_model: int,\n",
+    "        n_layer: int,\n",
+    "        input_dim: int,\n",
+    "        ssm_cfg=None,\n",
+    "        norm_epsilon: float = 1e-5,\n",
+    "        rms_norm: bool = False,\n",
+    "        initializer_cfg=None,\n",
+    "        fused_add_norm=False,\n",
+    "        residual_in_fp32=False,\n",
+    "        device=None,\n",
+    "        dtype=None,\n",
+    "    ) -> None:\n",
+    "        factory_kwargs = {\"device\": device, \"dtype\": dtype}\n",
+    "        super().__init__()\n",
+    "        self.residual_in_fp32 = residual_in_fp32\n",
+    "\n",
+    "        self.embedding = nn.Linear(input_dim, d_model, **factory_kwargs)\n",
+    "\n",
+    "        self.layers = nn.ModuleList(\n",
+    "            [\n",
+    "                create_block(\n",
+    "                    d_model,\n",
+    "                    ssm_cfg=ssm_cfg,\n",
+    "                    norm_epsilon=norm_epsilon,\n",
+    "                    residual_in_fp32=residual_in_fp32,\n",
+    "                    fused_add_norm=fused_add_norm,\n",
+    "                    layer_idx=i,\n",
+    "                    **factory_kwargs,\n",
+    "                )\n",
+    "                for i in range(n_layer)\n",
+    "            ]\n",
+    "        )\n",
+    "\n",
+    "        self.norm_f = nn.LayerNorm(d_model, eps=norm_epsilon, **factory_kwargs)\n",
+    "\n",
+    "        self.apply(\n",
+    "            partial(\n",
+    "                _init_weights,\n",
+    "                n_layer=n_layer,\n",
+    "                **(initializer_cfg if initializer_cfg is not None else {}),\n",
+    "            )\n",
+    "        )\n",
+    "\n",
+    "    def forward(self, x, inference_params=None, channel_last=False):\n",
+    "        if not channel_last:\n",
+    "            x = x.transpose(1, 2)\n",
+    "\n",
+    "        hidden_states = self.embedding(x)\n",
+    "        residual = None\n",
+    "        for layer in self.layers:\n",
+    "            hidden_states, residual = layer(\n",
+    "                hidden_states, residual, inference_params=inference_params\n",
+    "            )\n",
+    "\n",
+    "        residual = (hidden_states + residual) if residual is not None else hidden_states\n",
+    "        hidden_states = self.norm_f(residual.to(dtype=self.norm_f.weight.dtype))\n",
+    "\n",
+    "        hidden_states = hidden_states\n",
+    "\n",
+    "        return hidden_states\n",
+    "\n",
+    "    def representation(\n",
+    "        self,\n",
+    "        x: torch.Tensor,\n",
+    "        lengths: torch.Tensor,\n",
+    "        channel_last: bool = False,\n",
+    "    ) -> torch.Tensor:\n",
+    "        \"\"\"Get global representation of input data.\n",
+    "\n",
+    "        Args:\n",
+    "            x: Data to embed. Has shape (B x C x L) if not channel_last.\n",
+    "            lengths: Unpadded length of each data input.\n",
+    "            channel_last: Expects input of shape (B x L x C).\n",
+    "\n",
+    "        Returns:\n",
+    "            Global representation vector of shape (B x H).\n",
+    "        \"\"\"\n",
+    "        out = self.forward(x, channel_last=channel_last)\n",
+    "\n",
+    "        mean_tensor = mean_unpadded(out, lengths)\n",
+    "        return mean_tensor\n",
+    "\n",
+    "\n",
+    "def mean_unpadded(x: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:\n",
+    "    \"\"\"Take mean of tensor across second dimension without padding.\n",
+    "\n",
+    "    Args:\n",
+    "        x: Tensor to take unpadded mean. Has shape (B x L x H).\n",
+    "        lengths: Tensor of unpadded lengths. Has shape (B)\n",
+    "\n",
+    "    Returns:\n",
+    "        Mean tensor of shape (B x H).\n",
+    "    \"\"\"\n",
+    "    mask = torch.arange(x.size(1), device=x.device)[None, :] < lengths[:, None]\n",
+    "    masked_tensor = x * mask.unsqueeze(-1)\n",
+    "    sum_tensor = masked_tensor.sum(dim=1)\n",
+    "    mean_tensor = sum_tensor / lengths.unsqueeze(-1).float()\n",
+    "\n",
+    "    return mean_tensor\n",
+    "\n",
+    "\n",
+    "def _init_weights(\n",
+    "    module,\n",
+    "    n_layer,\n",
+    "    initializer_range=0.02,  # Now only used for embedding layer.\n",
+    "    rescale_prenorm_residual=True,\n",
+    "    n_residuals_per_layer=1,  # Change to 2 if we have MLP\n",
+    "):\n",
+    "    if isinstance(module, nn.Linear):\n",
+    "        if module.bias is not None:\n",
+    "            if not getattr(module.bias, \"_no_reinit\", False):\n",
+    "                nn.init.zeros_(module.bias)\n",
+    "    elif isinstance(module, nn.Embedding):\n",
+    "        nn.init.normal_(module.weight, std=initializer_range)\n",
+    "\n",
+    "    if rescale_prenorm_residual:\n",
+    "        for name, p in module.named_parameters():\n",
+    "            if name in [\"out_proj.weight\", \"fc2.weight\"]:\n",
+    "                nn.init.kaiming_uniform_(p, a=math.sqrt(5))\n",
+    "                with torch.no_grad():\n",
+    "                    p /= math.sqrt(n_residuals_per_layer * n_layer)\n",
+    "\n",
+    "def load_model(run_path: str, checkpoint_name: str) -> nn.Module:\n",
+    "    \"\"\"Load trained model located at specified path.\n",
+    "\n",
+    "    Args:\n",
+    "        run_path: Path where run data is located.\n",
+    "        checkpoint_name: Name of model checkpoint to load.\n",
+    "\n",
+    "    Returns:\n",
+    "        Model with loaded weights.\n",
+    "    \"\"\"\n",
+    "    model_config_path = os.path.join(run_path, \"model_config.json\")\n",
+    "    data_config_path = os.path.join(run_path, \"data_config.json\")\n",
+    "\n",
+    "    with open(model_config_path, \"r\") as f:\n",
+    "        model_params = json.load(f)\n",
+    "\n",
+    "    # TODO: Temp backwards compatibility\n",
+    "    if \"n_tracks\" not in model_params:\n",
+    "        with open(data_config_path, \"r\") as f:\n",
+    "            data_params = json.load(f)\n",
+    "        n_tracks = data_params[\"n_tracks\"]\n",
+    "    else:\n",
+    "        n_tracks = model_params[\"n_tracks\"]\n",
+    "\n",
+    "    model_path = os.path.join(run_path, checkpoint_name)\n",
+    "\n",
+    "    model = MixerModel(\n",
+    "        d_model=model_params[\"ssm_model_dim\"],\n",
+    "        n_layer=model_params[\"ssm_n_layers\"],\n",
+    "        input_dim=n_tracks\n",
+    "    )\n",
+    "    checkpoint = torch.load(model_path, map_location=torch.device('cpu'))\n",
+    "\n",
+    "    state_dict = {}\n",
+    "    for k, v in checkpoint[\"state_dict\"].items():\n",
+    "        if k.startswith(\"model\"):\n",
+    "            state_dict[k.lstrip(\"model\")[1:]] = v\n",
+    "\n",
+    "    model.load_state_dict(state_dict)\n",
+    "    return model\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "MixerModel(\n",
+      "  (embedding): Linear(in_features=6, out_features=512, bias=True)\n",
+      "  (layers): ModuleList(\n",
+      "    (0-5): 6 x Block(\n",
+      "      (mixer): Mamba(\n",
+      "        (in_proj): Linear(in_features=512, out_features=2048, bias=False)\n",
+      "        (conv1d): Conv1d(1024, 1024, kernel_size=(4,), stride=(1,), padding=(3,), groups=1024)\n",
+      "        (act): SiLU()\n",
+      "        (x_proj): Linear(in_features=1024, out_features=64, bias=False)\n",
+      "        (dt_proj): Linear(in_features=32, out_features=1024, bias=True)\n",
+      "        (out_proj): Linear(in_features=1024, out_features=512, bias=False)\n",
+      "      )\n",
+      "      (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
+      "    )\n",
+      "  )\n",
+      "  (norm_f): LayerNorm((512,), eps=1e-05, elementwise_affine=True)\n",
+      ")\n"
+     ]
+    }
+   ],
+   "source": [
+    "run_name=\"ssm_6_512_lr0.001_wd5e-05_mask0.15_seed0_splice_all_basic_eutheria_gene-dict\"\n",
+    "checkpoint=\"epoch=22-step=20000.ckpt\"\n",
+    "model_repository=\"/scratch/hdd001/home/phil/msk_backup/runs/\"\n",
+    "model = load_model(f\"{model_repository}{run_name}\", checkpoint_name=checkpoint)\n",
+    "model = model.to(torch.device('cuda'))\n",
+    "print(model)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[<Transcript ENST00000380152.7 of BRCA2>, <Transcript ENST00000544455.5 of BRCA2>, <Transcript ENST00000530893.6 of BRCA2>, <Transcript ENST00000614259.1 of BRCA2>, <Transcript ENST00000528762.1 of BRCA2>, <Transcript ENST00000470094.1 of BRCA2>, <Transcript ENST00000533776.1 of BRCA2>]\n",
+      "[11986, 10984, 2011, 7950, 495, 842, 523]\n",
+      "torch.Size([1, 6, 11986])\n",
+      "torch.Size([1, 512])\n"
+     ]
+    }
+   ],
+   "source": [
+    "transcripts = find_transcript_by_gene_name(genome, 'BRCA2')\n",
+    "print(transcripts)\n",
+    "print([sum(len(e) for e in t.exons) for t in transcripts])\n",
+    "t = transcripts[0]\n",
+    "sixt = create_six_track_encoding(t)\n",
+    "sixt = torch.tensor(sixt, dtype=torch.float32)\n",
+    "sixt = sixt.unsqueeze(0)\n",
+    "print(sixt.shape)\n",
+    "sixt = sixt.to(device='cuda')\n",
+    "lengths = torch.tensor([sixt.shape[2]]).to(device='cuda')\n",
+    "embedding = model.representation(sixt, lengths)\n",
+    "print(embedding.shape)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "MixerModel(\n",
+      "  (embedding): Linear(in_features=4, out_features=256, bias=True)\n",
+      "  (layers): ModuleList(\n",
+      "    (0-2): 3 x Block(\n",
+      "      (mixer): Mamba(\n",
+      "        (in_proj): Linear(in_features=256, out_features=1024, bias=False)\n",
+      "        (conv1d): Conv1d(512, 512, kernel_size=(4,), stride=(1,), padding=(3,), groups=512)\n",
+      "        (act): SiLU()\n",
+      "        (x_proj): Linear(in_features=512, out_features=48, bias=False)\n",
+      "        (dt_proj): Linear(in_features=16, out_features=512, bias=True)\n",
+      "        (out_proj): Linear(in_features=512, out_features=256, bias=False)\n",
+      "      )\n",
+      "      (norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
+      "    )\n",
+      "  )\n",
+      "  (norm_f): LayerNorm((256,), eps=1e-05, elementwise_affine=True)\n",
+      ")\n"
+     ]
+    }
+   ],
+   "source": [
+    "run_name=\"ssm_4t_3_256_lr0.001_wd1e-05_mask0.15_splice_two_none\"\n",
+    "checkpoint=\"epoch=83-step=14000.ckpt\"\n",
+    "model_repository=\"/scratch/hdd001/home/phil/msk_backup/grid_runs/\"\n",
+    "model = load_model(f\"{model_repository}{run_name}\", checkpoint_name=checkpoint)\n",
+    "model = model.to(torch.device('cuda'))\n",
+    "print(model)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "torch.Size([1, 4, 495])\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "torch.Size([1, 256])"
+      ]
+     },
+     "execution_count": 43,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "seq = \"\".join([genome.dna(exon) for exon in transcripts[4].exons])\n",
+    "one_hot = seq_to_oh(seq)\n",
+    "one_hot = one_hot.T\n",
+    "torch_one_hot = torch.tensor(one_hot, dtype=torch.float32)\n",
+    "torch_one_hot = torch_one_hot.unsqueeze(0)\n",
+    "print(torch_one_hot.shape)\n",
+    "torch_one_hot = torch_one_hot.to(device='cuda')\n",
+    "lengths = torch.tensor([torch_one_hot.shape[2]]).to(device='cuda')\n",
+    "model.representation(torch_one_hot, lengths).shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "torch_rna",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

src/model.py

@@ -0,0 +1,205 @@
+import math
+from functools import partial
+import os
+import json
+import torch
+import torch.nn as nn
+
+from mamba_ssm.modules.mamba_simple import Mamba, Block
+from huggingface_hub import PyTorchModelHubMixin
+
+def create_block(
+    d_model,
+    ssm_cfg=None,
+    norm_epsilon=1e-5,
+    residual_in_fp32=False,
+    fused_add_norm=False,
+    layer_idx=None,
+    device=None,
+    dtype=None,
+):
+    if ssm_cfg is None:
+        ssm_cfg = {}
+    factory_kwargs = {"device": device, "dtype": dtype}
+    mix_cls = partial(Mamba, layer_idx=layer_idx, **ssm_cfg, **factory_kwargs)
+    norm_cls = partial(nn.LayerNorm, eps=norm_epsilon, **factory_kwargs)
+    block = Block(
+        d_model,
+        mix_cls,
+        norm_cls=norm_cls,
+        fused_add_norm=fused_add_norm,
+        residual_in_fp32=residual_in_fp32,
+    )
+    block.layer_idx = layer_idx
+    return block
+
+
+class MixerModel(
+    nn.Module,
+    PyTorchModelHubMixin,
+):
+
+    def __init__(
+        self,
+        d_model: int,
+        n_layer: int,
+        input_dim: int,
+        ssm_cfg=None,
+        norm_epsilon: float = 1e-5,
+        rms_norm: bool = False,
+        initializer_cfg=None,
+        fused_add_norm=False,
+        residual_in_fp32=False,
+        device=None,
+        dtype=None,
+    ) -> None:
+        factory_kwargs = {"device": device, "dtype": dtype}
+        super().__init__()
+        self.residual_in_fp32 = residual_in_fp32
+
+        self.embedding = nn.Linear(input_dim, d_model, **factory_kwargs)
+
+        self.layers = nn.ModuleList(
+            [
+                create_block(
+                    d_model,
+                    ssm_cfg=ssm_cfg,
+                    norm_epsilon=norm_epsilon,
+                    residual_in_fp32=residual_in_fp32,
+                    fused_add_norm=fused_add_norm,
+                    layer_idx=i,
+                    **factory_kwargs,
+                )
+                for i in range(n_layer)
+            ]
+        )
+
+        self.norm_f = nn.LayerNorm(d_model, eps=norm_epsilon, **factory_kwargs)
+
+        self.apply(
+            partial(
+                _init_weights,
+                n_layer=n_layer,
+                **(initializer_cfg if initializer_cfg is not None else {}),
+            )
+        )
+
+    def forward(self, x, inference_params=None, channel_last=False):
+        if not channel_last:
+            x = x.transpose(1, 2)
+
+        hidden_states = self.embedding(x)
+        residual = None
+        for layer in self.layers:
+            hidden_states, residual = layer(
+                hidden_states, residual, inference_params=inference_params
+            )
+
+        residual = (hidden_states + residual) if residual is not None else hidden_states
+        hidden_states = self.norm_f(residual.to(dtype=self.norm_f.weight.dtype))
+
+        hidden_states = hidden_states
+
+        return hidden_states
+
+    def representation(
+        self,
+        x: torch.Tensor,
+        lengths: torch.Tensor,
+        channel_last: bool = False,
+    ) -> torch.Tensor:
+        """Get global representation of input data.
+
+        Args:
+            x: Data to embed. Has shape (B x C x L) if not channel_last.
+            lengths: Unpadded length of each data input.
+            channel_last: Expects input of shape (B x L x C).
+
+        Returns:
+            Global representation vector of shape (B x H).
+        """
+        out = self.forward(x, channel_last=channel_last)
+
+        mean_tensor = mean_unpadded(out, lengths)
+        return mean_tensor
+
+
+def mean_unpadded(x: torch.Tensor, lengths: torch.Tensor) -> torch.Tensor:
+    """Take mean of tensor across second dimension without padding.
+
+    Args:
+        x: Tensor to take unpadded mean. Has shape (B x L x H).
+        lengths: Tensor of unpadded lengths. Has shape (B)
+
+    Returns:
+        Mean tensor of shape (B x H).
+    """
+    mask = torch.arange(x.size(1), device=x.device)[None, :] < lengths[:, None]
+    masked_tensor = x * mask.unsqueeze(-1)
+    sum_tensor = masked_tensor.sum(dim=1)
+    mean_tensor = sum_tensor / lengths.unsqueeze(-1).float()
+
+    return mean_tensor
+
+
+def _init_weights(
+    module,
+    n_layer,
+    initializer_range=0.02,  # Now only used for embedding layer.
+    rescale_prenorm_residual=True,
+    n_residuals_per_layer=1,  # Change to 2 if we have MLP
+):
+    if isinstance(module, nn.Linear):
+        if module.bias is not None:
+            if not getattr(module.bias, "_no_reinit", False):
+                nn.init.zeros_(module.bias)
+    elif isinstance(module, nn.Embedding):
+        nn.init.normal_(module.weight, std=initializer_range)
+
+    if rescale_prenorm_residual:
+        for name, p in module.named_parameters():
+            if name in ["out_proj.weight", "fc2.weight"]:
+                nn.init.kaiming_uniform_(p, a=math.sqrt(5))
+                with torch.no_grad():
+                    p /= math.sqrt(n_residuals_per_layer * n_layer)
+
+def load_model(run_path: str, checkpoint_name: str) -> nn.Module:
+    """Load trained model located at specified path.
+
+    Args:
+        run_path: Path where run data is located.
+        checkpoint_name: Name of model checkpoint to load.
+
+    Returns:
+        Model with loaded weights.
+    """
+    model_config_path = os.path.join(run_path, "model_config.json")
+    data_config_path = os.path.join(run_path, "data_config.json")
+
+    with open(model_config_path, "r") as f:
+        model_params = json.load(f)
+
+    # TODO: Temp backwards compatibility
+    if "n_tracks" not in model_params:
+        with open(data_config_path, "r") as f:
+            data_params = json.load(f)
+        n_tracks = data_params["n_tracks"]
+    else:
+        n_tracks = model_params["n_tracks"]
+
+    model_path = os.path.join(run_path, checkpoint_name)
+
+    model = MixerModel(
+        d_model=model_params["ssm_model_dim"],
+        n_layer=model_params["ssm_n_layers"],
+        input_dim=n_tracks
+    )
+    checkpoint = torch.load(model_path, map_location=torch.device('cpu'))
+
+    state_dict = {}
+    for k, v in checkpoint["state_dict"].items():
+        if k.startswith("model"):
+            state_dict[k.lstrip("model")[1:]] = v
+
+    model.load_state_dict(state_dict)
+    return model