How did this guy train his model?

username: flyingpower
cooked: <p>So, basically i have been watching a specific project for old captcha solving, and i would like to know how he trained it (because he did not say how he made it)</p>
<p>his input is:<br>
Class (lets say, the captcha wants a grape, you set that as the input so the model know what to check for, look examples below)<br>
Pictures (the captcha gives 4 slightly deformed pictures and one of them is a grape) (it’s pixelart)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da58c3614bfa7ea347a1f3c4fc41a470b765375e.png" data-download-href="https://discuss.pytorch.org/uploads/default/da58c3614bfa7ea347a1f3c4fc41a470b765375e" title="Unbenannt"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da58c3614bfa7ea347a1f3c4fc41a470b765375e_2_690x388.png" alt="Unbenannt" data-base62-sha1="v9Agin6c9l9Sz2Nue0WZpqsvVdc" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da58c3614bfa7ea347a1f3c4fc41a470b765375e_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da58c3614bfa7ea347a1f3c4fc41a470b765375e_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da58c3614bfa7ea347a1f3c4fc41a470b765375e_2_1380x776.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Unbenannt</span><span class="informations">1920×1080 42.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How did he do that?</p>
<p>his code:<br>
verifier=</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import utils
from model.resnet import resnet34
import torchvision.transforms as transforms
import os
import time
from torch.autograd import Variable
from PIL import Image

model_dir = './output/model.pt'
    
DEVICE = torch.device('cpu')
NUM_classes = 40

CLASS_NAMES = ["banana", "book", "bread", "candy cane", "candy corn", "cannon", "carrot", "cheese", "cherry", "chest piece", "clock", "diamond",
               "egg", "fire", "fish", "frog", "ghost", "grapes", "gun", "hat", "helmet", "house", "key", "lemon", "mushroom", "necklace", "pear", 
               "pepper", "pie", "piece of meat", "pineapple", "pretzel", "pumpkin", "rose", "strawberry", "treasure chest", "watermelon", "empty bottle", "orange", "crown"]
               

def preprocessing_image(img, transform):

    if len(img.split()) == 4:
        r, g, b, a = img.split()
    elif len(img.split()) == 3:
        r, g, b = img.split()
    img = Image.merge("RGB", (r, g, b))
    img = img.resize((224, 224), Image.ANTIALIAS)
    img = transform(img)
    img = img.view(-1, 3, 224, 224)
    return img

def simmmover(image_path, item):

    real = utils.getLabelValue(item)
    
    transformations = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])

    
    model = resnet34(pretrained=False, num_classes=NUM_classes)
    model.load_state_dict(torch.load(model_dir))
    model = model.to(DEVICE)
    
    transformations = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])
    model.eval()
    
    img1 = preprocessing_image(Image.open(os.path.join(image_path, 'img1.png')), transformations)
    img2 = preprocessing_image(Image.open(os.path.join(image_path, 'img2.png')), transformations)
    img3 = preprocessing_image(Image.open(os.path.join(image_path, 'img3.png')), transformations)
    img4 = preprocessing_image(Image.open(os.path.join(image_path, 'img4.png')), transformations)
    
    #Looks for the most accurate result, given its name &amp; pictures if the picture matches with the name -&gt; result
    img1, img2, img3, img4 = img1.to(DEVICE), img2.to(DEVICE), img3.to(DEVICE), img4.to(DEVICE)
    
    prediction1 = model(img1).squeeze()
    prediction2 = model(img2).squeeze()
    prediction3 = model(img3).squeeze()
    prediction4 = model(img4).squeeze()
    print(prediction1[real])
    print(prediction2[real])
    print(prediction3[real])
    print(prediction4[real])
    #_, predicted1 = torch.max(prediction1, 1)
    #_, predicted2 = torch.max(prediction2, 1)
    #_, predicted3 = torch.max(prediction3, 1)
    #_, predicted4 = torch.max(prediction4, 1)
    _, predicted = torch.max(torch.tensor([prediction1[real], prediction2[real], prediction3[real], prediction4[real]]), 0)
    result = 'img' + str(predicted.item()+1)
    return result
   
</code></pre>
<p>example1=</p>
<pre><code class="lang-auto">                element = driver.find_element(By.CSS_SELECTOR, ".text-2xl")
                item = element.text.lower()
                prediction = verifier.simmmover("./temporary", item)

                if prediction == 'img1':
                    es1.click()
                elif prediction == 'img2':
                    es2.click()
                elif prediction == 'img3':
                    es3.click()
                elif prediction == 'img4':
                    es4.click()
            
                time.sleep(3)
</code></pre>
<p>example 2=</p>
<pre><code class="lang-auto">item = "grapes"
prediction = verifier_test.simmmover("./temporary", item)
print(prediction)
</code></pre>
<p>(temporary = the images from the given captcha)</p>

username: flyingpower
cooked: <p>Also i’m new to this, any code example is welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: eqy
cooked: <p>This looks like it is producing unnormalized logits for a multiclass classification problem. I would take a look at e.g., the MNIST or ImageNet examples to see examples of training these types of vision models.<br>
Note that even if the examples are outputting a prediction class, you can still extract the raw logit/softmax value to rank the images in terms of “confidence” for a target class such as “grape.”<br>
MNIST:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/examples/tree/main/mnist">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/examples/tree/main/mnist" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/examples/tree/main/mnist" target="_blank" rel="noopener nofollow ugc">examples/mnist at main · pytorch/examples</a></h3>

  <p><a href="https://github.com/pytorch/examples/tree/main/mnist" target="_blank" rel="noopener nofollow ugc">main/mnist</a></p>

  <p><span class="label1">A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
ImageNet:</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/examples/tree/main/imagenet">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/examples/tree/main/imagenet" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/examples/tree/main/imagenet" target="_blank" rel="noopener nofollow ugc">examples/imagenet at main · pytorch/examples</a></h3>

  <p><a href="https://github.com/pytorch/examples/tree/main/imagenet" target="_blank" rel="noopener nofollow ugc">main/imagenet</a></p>

  <p><span class="label1">A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>