Convert Text data to type which can be standartized

Tornike: <p>Hi, i want to convert text data where it will be possible to standardize it. This is my Desired Pipiline:</p>
<p>1)convert data to the way it will be possible to standardize.<br>
2)apply standardized data to Linear Discriminant Analysis for Dimensionality Reduction.<br>
3)After LDA apply it to CountVectorizer<br>
4)then split and train with Traditional ML Algorithm.</p>
<p>Is it a correct pipeline? Right now i’m stuck at the first step. how Should i convert from string to digits that it will be possible to standardize?</p>
<p>Or may be there is an easier way to do that with different pipeline ?</p>
<p>All tutorials are on IRIS dataset, where every column is continues digits data. not text.</p>

J_Johnson: <p>Have you tried this tutorial?</p>
<p><a href="https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html</a></p>
<p>By the way, I found one of the definitions should be changed here:</p>
<pre><code class="lang-auto">from torchtext.vocab import vocab # note lower case "vocab"

def build_vocab(filepath, tokenizer):
  counter = Counter()
  with io.open(filepath, encoding="utf8") as f:
    for string_ in f:
      counter.update(tokenizer(string_))
  return vocab(counter, specials=['&lt;unk&gt;', '&lt;pad&gt;', '&lt;bos&gt;', '&lt;eos&gt;'])  # note lower case "vocab"
</code></pre>

J_Johnson: <p>Alternatively, you can use <code>generate_sp_model</code>:</p>
<p><a href="https://pytorch.org/text/stable/data_functional.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/text/stable/data_functional.html</a></p>
<p>That will give you a way to produce an efficient sentencepiece model and go back and forth between strings and tokens.</p>
<p>Here is the original paper:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1808.06226">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1808.06226" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/1808.06226" target="_blank" rel="noopener nofollow ugc">SentencePiece: A simple and language independent subword tokenizer and...</a></h3>

  <p>This paper describes SentencePiece, a language-independent subword tokenizer
and detokenizer designed for Neural-based text processing, including Neural
Machine Translation. It provides open-source C++ and Python implementations for
subword units....</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Tornike: <p>Thanks i’ll check those.</p>

Tornike: <p>One question though, while training the ML model with Kfold cross-validation. Is it okay to dump trained model at every SKF split to use that in testing ?  or how do i use Kfold trained model? i want Earlystopping in Traditional ML</p>

J_Johnson: <p>Are you limited on data? Having a dedicated validation set would be better than Kfold, imo, if not.</p>
<p>I’m not sure what you mean by “dump” the model.</p>
<p>It’s good practice to make checkpoints between each epoch and a log of any running stats. Then you can choose which set of parameters were best after the fact.</p>

J_Johnson: <p>When you want to run validation data, just be sure to use these bits of code:</p>
<pre><code class="lang-auto">
with torch.no_grad():
    model.eval()
    #test and get loss or other stats here
    #good spot to make a checkpoint with stats log
    model.train()

#continue training

</code></pre>

Tornike: <p>Yes, i meant checkpoints. but i did the project in Traditional ML. so that’s why i asked if it’s ok to save checkpoints after every k-fold training.</p>
<p>Thank you so much. I already finished that project.</p>