Talk:Stochastic gradient descent

message: Could someone please diamb. the optimisation link? I'd love to do it, but I cannot figure if this would have to be the mathematical optimisation (finding the min-max or balance of a function) or the computer sciences optimisation (making a program ressource-efficient)... --Tribaal 08:26, 12 January 2006 (UTC)
thread_title: Optimisation?

message: I'm currently attempting to implement a stochastic hill-climbing algorithm into code, and I still don't know what the heck this page is on about. Could we perhaps edit it so the words actually mean something to someone? —Preceding unsigned comment added by 68.118.0.110
: You may read Stochastic hill climbing. I think these two terms refer to different things. 129.186.93.219 (talk) 23:47, 15 April 2009 (UTC)
thread_title: Does this article text actually mean anything?

message: Could it not be said the the usual method of training a back propagation neural network (i.e., by picking a random training element at each step and adjusting the weight matrix based on the error it produces) is an instance of this algorithm?  And probably by far the most common one? 92.41.75.253 (talk) 14:33, 24 October 2008 (UTC)
thread_title:  Back propagation training algorith? 

message: In the definition of the gradient descent algorithm there is no definition for the quantity delta.  — Preceding unsigned comment added by Eep1mp (talk • contribs) 02:28, 9 February 2011 (UTC)
thread_title:  Delta 

message: Introduces an "x-prime" term without defining it, nor is it obvious what it should mean.Dmumme (talk) 21:17, 12 March 2019 (UTC)
thread_title:  Implicit updates (ISGD) 

message: Isn't the example in this article talking about batch gradient descent? Could someone please clarify?  — Preceding unsigned comment added by Kindlychung (talk • contribs) 09:51, 13 April 2014 (UTC)
thread_title:  Example section 

message: : w := w - \alpha \nabla Q(w) = w - \alpha \sum_{i=1}^n \nabla Q_i(w),
I am a little confused here, the sum of gradients is a scalar, and w is a vector of coefficients, how can you subtract the one from the other?  — Preceding unsigned comment added by Kindlychung (talk • contribs) 09:55, 13 April 2014 (UTC)
replies: text: Gradient is a vector (http://en.wikipedia.org/wiki/Gradient), so sum of gradients is also a vector. 213.33.220.118 (talk) 12:35, 4 June 2014 (UTC)
thread_title:  Formula in the background section 

message: The article does not explain l2 regularization, which is a commonly used adaptation of the algorithm that improves convergence and keeps weights from exploding. Regularization in online optimization settings is an interesting topic on its own right, so it would be good to cover it here.
I'm working on adding a section for regularization to this article in my sandbox.  — Preceding unsigned comment added by Groceryheist (talk • contribs) 16:07, 11 April 2016 (UTC)
thread_title:  Regularization 

message: As per the paper (https://arxiv.org/abs/1412.6980), Algorithm 1 on page 2, the update rule for Adam uses bias-corrected versions of the first and second moment estimates, m and v. The formulas in the article are missing the divisions by 1-\beta_1^t and 1-\beta_2^t.  — Preceding unsigned comment added by Andersgb1 (talk • contribs) 10:34, 14 June 2016 (UTC)
thread_title:  Error in formulas for Adam 

message: The first citation in this article, and the only citation in the introduction is to a 2018 paper of Mei et al.  Is this really appropriate?  I have looked at the paper.  It is of some specialized interest (although I can't say one way or the other whether it is technically correct), but it is hardly highly important, nor is it central to the SGD algorithm or its history.  (By way of comparison, the original Robbins-Monro does not have a formal citation.)
I have seen citations to Mei et al. pop up in unexpected places.  I think there is a possibility of this citation being academic spam.  Would it make sense to check when this citation was added, and by whom?
: No, it isn't. I edited the introduction to include citations that are much more relevant and established. Now citing the paper of Robbins-Monro and the paper by Tong Zhang (2004) which kickstarted the use of SGD in machine learning. Before there were citations to Mei et al and a Business textbook, which were not as relevant. (perhaps academic spamming).--Magicheader (talk) 22:43, 25 November 2019 (UTC)
thread_title:  Is citing the Mei et al. paper appropriated? 

message: Don't understand the maths 2603:8001:8100:C71D:50E4:A0EA:F266:835F (talk) 12:14, 4 April 2022 (UTC)
replies: text: I agree that the structure of the article doesn't currently support any sort of 'gentle' or more generally helpful explanation, such as an analogy or salient visual. Does anyone have any ideas on how we might better structure it? I think a higher-level introduction would be better to start with, but I have no opinion on where the history section should go. VirtualVistas (talk) 18:09, 15 October 2023 (UTC)
thread_title:  Start at the beginning