How to pass separate features into LSTM

username: CangyuanLi
cooked: <p>Hi, I am new to PyTorch (and machine learning in general) and wanted to check if what I’m doing makes any sense at all. I am attempting to predict race (Asian, Black, Hispanic, White) from first name, last name, and the racial distribution of the person’s zip code. These should be three separate features, the thought being that, e.g. last name might have more predictive power than first name. For example, a row of my data would be</p>
<pre><code class="lang-auto">("John", "Li", [0.10, 0.40, 0.20, 0.30])
</code></pre>
<p>Currently, my model is</p>
<pre><code class="lang-auto">class FirstLastZctaLSTM(nn.Module):
    def __init__(self, input_size: int, hidden_size: int, output_size: int) -&gt; None:
        super(FirstLastZctaLSTM, self).__init__()

        self.hidden_size = hidden_size
        self.ltsm_cell = nn.LSTM(input_size, hidden_size)
        self.h2o = nn.Linear(hidden_size + 4, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(
        self,
        name: torch.Tensor,
        pct: torch.Tensor,
        hidden: tuple[torch.Tensor, torch.Tensor],
    ):
        _, hidden = self.ltsm_cell(name.view(1, 1, -1), hidden)
        combined = torch.cat([hidden[0].squeeze(0), pct], dim=1)
        output = self.h2o(combined)
        output: torch.Tensor = self.softmax(output)

        return output, hidden

    def init_hidden(self):
        return (
            torch.zeros(1, 1, self.hidden_size, device=DEVICE),
            torch.zeros(1, 1, self.hidden_size, device=DEVICE),
        )

</code></pre>
<p>I use a character-level encoding</p>
<pre><code class="lang-auto">import string
VALID_NAME_CHARS = f"{string.ascii_lowercase} '-"
VALID_NAME_CHARS_DICT = {c: i for i, c in enumerate(VALID_NAME_CHARS)}
VALID_NAME_CHARS_LEN = len(VALID_NAME_CHARS)

def encode_name(
    name: str,
    valid_name_chars_dict: dict[str, int],
    valid_name_chars_len: int,
    device: torch.device,
) -&gt; torch.Tensor:
    encoded = torch.zeros(len(name), 1, valid_name_chars_len, device=device)
    for idx, c in enumerate(name):
        encoded[idx][0][valid_name_chars_dict[c]] = 1

    return encoded
</code></pre>
<p>Then I combine the encoded tensors like so:</p>
<pre><code class="lang-auto">name = torch.cat([encode_name(first_name), encode_name(last_name)], dim=0)
</code></pre>
<p>As an example, for data “michael kitts”, the final name tensor is of size [12, 29] and the percent data looks like</p>
<pre><code class="lang-auto">tensor([[6.0413e-03, 8.2485e-04, 3.7458e-02, 9.0547e-01]])
</code></pre>
<p>I pass it into the model</p>
<pre><code class="lang-auto">for name, pct, race in dataloader:
    name = name.squeeze()  # dataloader adds an extra batch size dimension
    model.zero_grad(set_to_none=True)
    hidden = model.init_hidden()
    for i in range(name.size()[0]):
        output, hidden = model(name[i], pct, hidden)
</code></pre>
<p>It runs with no errors, but I am curious if this is doing what I want it be doing? Since I concatenate first and last name, does it mean that I am basically just passing in the full name, just without the space?</p>
<p>Thanks a lot.</p>

username: vdw
cooked: <p>You code is a bit confusing to read, so maybe I’m got some things wrong.</p>
<p>You’re using an <code>nn.LSTM</code> instead of <code>nn.LSTMCell</code>, but you treat it as a cell since you give it only one character at a time as input; hence, the need for a loop that iterates over all characters in the name. You can give an <code>nn.LSTM</code> only one character at the time, but it is a bit odd.</p>
<p>It’s unclear what <code>pct</code> is, so I assume it’s the tensor reflecting the race distribution you concatenate with the current hidden state. According to your code, you concatenate <code>pct</code> at <strong>each</strong> time step. You can do this, but it does not seem intuitive. <code>pct</code> is a single vector associated with your while training sample, i.e., the name, and not the individual characters. In other words, <code>pct</code> itself is not sequential in nature. The more intuitive solution would to only feed the name to the LSTM and concatenate only the last hidden state with <code>pct</code>.</p>

username: CangyuanLi
cooked: <p>Thanks a lot for the response. Sorry for the confusion! You are right about what <code>pct</code> is. Regarding passing each character, I was attempting to adapt <a href="https://www.askpython.com/python/examples/predict-nationality-using-names" class="inline-onebox" rel="noopener nofollow ugc">Predict Nationality Based On Name In Python - AskPython</a>. Does a model like this make more sense then? Now the entire name is passed to lstm, so when I combine it with the pct vector it is equivalent to only using it at the last hidden state?</p>
<pre><code class="lang-auto">class FirstLastZctaLSTM(nn.Module):
    def __init__(self, input_size: int, hidden_size: int, output_size: int) -&gt; None:
        super(FirstLastZctaLSTM, self).__init__()

        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size)
        self.h2o = nn.Linear(hidden_size + 4, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(
        self,
        name: torch.Tensor,
        pct: torch.Tensor,
        hidden: tuple[torch.Tensor, torch.Tensor],
    ):
        _, hidden = self.lstm(name, hidden)
        combined = torch.cat([hidden[0].squeeze(0), pct], dim=1)
        output = self.h2o(combined)
        output: torch.Tensor = self.softmax(output)

        return output, hidden

    def init_hidden(self):
        return (
            torch.zeros(1, 1, self.hidden_size, device=DEVICE),
            torch.zeros(1, 1, self.hidden_size, device=DEVICE),
        )
</code></pre>

username: vdw
cooked: <p>For predicting the nationality based on names, there’s actually a <a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="noopener nofollow ugc">PyTorch tutorial</a> you can have a look at as well.</p>
<p>Note that <code>hidden[0]</code> only works here because you initialize the <code>nn.LSTM</code> with the default parameters <code>num_layers=1</code> and <code>bidirectional=False</code>. For example <code>hidden[-1]</code> is a bit cleaner as it handles the case where you want to use more than 1 layer. If you want to use a bidirectional LSTM, you need to be a tad more careful.</p>
<p>That being said, concatenating <code>pct</code> to the last hidden state seems more intuitive to me. Whether the results are better or worse is a different story, but that’s how I would approach this task. But with neural networks, in general, there are many alternatives. For example you could push the last hidden state first through an additional linear layer (e.g., to reduce the dimension) before concatenating it with <code>pct</code>. Which “makes more sense” is difficult to answer :).</p>

username: ashna_singh
cooked: <p>This post was flagged by the community and is temporarily hidden.</p>
cooked_hidden: yes
hidden_reason_id: 3