How do i use svm in nn?

username: yannick
cooked: <p>are there some ways to combine statastical learning model within nn? if we want to use BP to train the model, do we have to implement classical statestical model by pytorch?</p>

username: zzuczy
cooked: <p>scikit-learn may be what you want.<br>
<a href="http://scikit-learn.org/stable/" rel="nofollow noopener">http://scikit-learn.org/stable/</a></p>

username: yannick
cooked: <aside class="quote no-group quote-post-not-found" data-username="yannick" data-post="3" data-topic="10686" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/b782af/48.png" class="avatar"> yannick:</div>
<blockquote>
<p>but how do i get the gradient of the svm if we use it as the last layer or a block?</p>
</blockquote>
</aside>
<p>but how do i get the gradient of the svm if we use it as the last layer or a block?</p>

username: zzuczy
cooked: <p>I know what’s your mean now. I have to admit that I haven’t seen anyone do this before. As far as I know, SVM can only change its hyperplane without changing the input features. In other words，it only assumes that its hyperplane is not perfect and would not think that the input was wrong.<br>
We can think it in a different way. Suppose that SVM has the ability to change the representation of sample, which we call it as feats, we do not need DNN any more.<img src="//discuss.pytorch.org/images/emoji/apple/joy.png?v=5" title=":joy:" class="emoji" alt=":joy:"><br>
Sorry for not being able to help you.Maybe you can try other loss functions except Softmax.</p>

username: SimonW
cooked: <p>l1-svm is not differentiable, but l2-svm is. If you want to use it, you will need to implement it and the gradient calculation (sec. 2.4 of paper below) by yourself.</p>
<p>Ref: <a href="http://deeplearning.net/wp-content/uploads/2013/03/dlsvm.pdf">http://deeplearning.net/wp-content/uploads/2013/03/dlsvm.pdf</a></p>

username: Liang
cooked: <p>There might be a substitution for SVM, that is, a SVM-like loss. This paper holds the opinion that minimizing their loss at the last layer is analogous to minimizing the margin in an SVM classifier. (See <a href="https://pdfs.semanticscholar.org/8080/f1545004a598967b6d229c281a93787eaa1c.pdf" rel="nofollow noopener">this paper</a> sec.3.1 above, equation (3)). It looks like a hinge loss.</p>
<p>Sorry that I don’t know much about it. And the author did not give a theoretical analysis. You can try the loss if you want to.</p>

username: gabrieldernbach
cooked: <p>Citing the sklearn implementation of the stochastic gradient descent regressor - you would have to use the hinge loss</p>
<p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html" class="onebox" target="_blank" rel="nofollow noopener">https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html</a><br>
parameter: loss - Defaults to ‘hinge’, which gives a linear SVM.</p>
<p>SVM 0-1 loss has infinite slope at 0, the hinge loss can be seen as a differentiable approximation to it.</p>
<p>For further details see<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://math.stackexchange.com/questions/327807/why-hinge-loss-is-equivalent-to-0-1-loss-in-svm" target="_blank" rel="nofollow noopener">math.stackexchange.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://math.stackexchange.com/users/53766/tina" target="_blank" rel="nofollow noopener">
    <img alt="Tina" src="https://www.gravatar.com/avatar/afc6518086e1936c158fe1a4155bd513?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://math.stackexchange.com/questions/327807/why-hinge-loss-is-equivalent-to-0-1-loss-in-svm" target="_blank" rel="nofollow noopener">Why "hinge" loss is equivalent to 0-1 loss in SVM?</a>
</h4>

<div class="tags">
  <strong>statistics, self-learning, machine-learning</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://math.stackexchange.com/users/53766/tina" target="_blank" rel="nofollow noopener">
    Tina
  </a>
  on <a href="https://math.stackexchange.com/questions/327807/why-hinge-loss-is-equivalent-to-0-1-loss-in-svm" target="_blank" rel="nofollow noopener">09:23PM - 11 Mar 13 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>