I'm **solving a simplified logical regression problem**, that only uses **weights w1 and w2** and **no bias term**. The **data used** here is **generated synthetically to nicely fit into such model**. (Not having a bias terms, reduces the parameters  to update during training to w1 and w2, making it possible for me to visualise the entire gradient descent process with only 3 dimension axes later on).

# Issue: doubtful about Adam optimiser’s behaviour

I am **highly doubtful that my implementation is correctly reflecting behaviour of the Adam** optimiser. It is **taking very tiny steps**, and when setting **learning rate 0.02 it takes around 130 epochs to converge (it will stay at low accuracy near 20%, until around 123th epoch, then make big jumps in the coming three epochs** to reach near 80% accuracy).

To my knowledge Adam optimiser is generally set to lower learning rates like 0.001, and there onwards it automatically adjusts the learning rate to more reasonable rate. But **if I were to set my learning rate to 0.001, I’d need around 2536 epochs,** just for a simple logistic regression problem with only two weights.

I **feel like if Adam was correctly implemented** here, it **would have simply adjusted the learning rate during the training to converge much faster.**

# Code

**Gradient Descent Class**

```python
class GradientDescent:
    def __init__(self, optimizer=None):
        self.weights = None
        self.learning_rate = None
        self.optimizer = optimizer

    def fit(self, X_train, y_train, X_val, y_val, epochs=100, initial_weights=None, learning_rate=0.002, batch_size=32, verbose=True):
        self.learning_rate = learning_rate

        # Initialize lists to store metrics for each epoch
        train_losses, train_accuracies, val_losses, val_accuracies, weight_history = [], [], [], [], []

        for epoch in range(epochs):
            if epoch == 0:
                np.random.seed(42)
                self.weights = np.array(initial_weights) if initial_weights else np.random.uniform(-1, 1, num_features)
            else:
                for i in range(0, X_train.shape[0], batch_size):
                    X_batch = X_train[i:i + batch_size]
                    y_true_batch = y_train[i:i + batch_size]
    
                    # Compute linear combination and apply sigmoid
                    linear_combination = np.dot(X_batch, self.weights)
                    y_pred_batch = 1 / (1 + np.exp(-linear_combination))
    
                    # Compute gradient of binary-crossentropy loss w.r.t weights 
                    grad_loss = np.dot(X_batch.T, (y_pred_batch - y_true_batch)) / X_batch.shape[0]
    
                    # Apply optimizer 
                    if self.optimizer:
                        grad_loss = self.optimizer.update(grad_loss)
    
                    # Update weights
                    self.weights -= self.learning_rate * grad_loss

            # Training evaluation after the epoch
            train_loss, train_accuracy = self.evaluate(X_train, y_train)
            val_loss, val_accuracy = self.evaluate(X_val, y_val)

            # Store metrics & display the metrics for the current epoch
            weight_history.append(self.weights.copy()); train_losses.append(train_loss); train_accuracies.append(train_accuracy); val_losses.append(val_loss); val_accuracies.append(val_accuracy)
            if verbose:
                print(f"Epoch {epoch + 1}:\nWeights: {self.weights}\nTrain Loss: {train_loss:.8f}, Train Accuracy: {train_accuracy:.4f}\nValidation Loss: {val_loss:.8f}, Validation Accuracy: {val_accuracy:.4f}\n")

        return train_losses, train_accuracies, val_losses, val_accuracies, weight_history

    def evaluate(self, X, y_true):
        linear_combination = np.dot(X, self.weights)
        y_pred = 1 / (1 + np.exp(-linear_combination))
        loss = -np.mean(y_true * np.log(y_pred + 1e-8) + (1 - y_true) * np.log(1 - y_pred + 1e-8)) # Cross-entropy loss
        accuracy = np.mean((y_pred >= 0.5) == y_true)
        return loss, accuracy
```

**Adam Optimizer class**

```python
class Adam:
    def __init__(self, beta1=0.9, beta2=0.999, epsilon=1e-7):
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.m_t = None  # First moment estimate (momentum)
        self.v_t = None  # Second moment estimate (RMSProp-like)
        self.t = 0       # Timestep for bias correction

    def update(self, gradient):
        if self.m_t is None:
            self.m_t = np.zeros_like(gradient)
        if self.v_t is None:
            self.v_t = np.zeros_like(gradient)

        self.t += 1

        # Update biased first and second moment estimates
        self.m_t = self.beta1 * self.m_t + (1 - self.beta1) * gradient
        self.v_t = self.beta2 * self.v_t + (1 - self.beta2) * (gradient ** 2)

        # Bias-corrected moment estimates
        m_hat = self.m_t / (1 - self.beta1 ** self.t)
        v_hat = self.v_t / (1 - self.beta2 ** self.t)

        # Update weights
        adjusted_gradient = m_hat / (np.sqrt(v_hat) + self.epsilon)

        return adjusted_gradient
```

**Performing Gradient Descent**

```python
# Generate Synthetic data
np.random.seed(42)
true_weights = np.array([-3, 3.5])
X = np.random.rand(500, len(true_weights)) * 10
y = (1 / (1 + np.exp(-np.dot(X, true_weights))) > 0.5).astype(int)
y[np.random.choice(500, int(0.1 * 500), replace=False)] ^= 1

# Split the data into training (60%), validation (20%), and test (20%) sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_test, y_test, test_size=0.5, random_state=42)

# Training
gd = GradientDescent(optimizer=Adam())

# slowly converges at around 90th epoch
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=100, batch_size=32, initial_weights=[15.0, -20.0], learning_rate=0.03)

# slowly converges at around 130th epoch
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=500, batch_size=32, initial_weights=[15.0, -20.0], learning_rate=0.02)

# slowly converges at around 2536th epoch
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=5000, batch_size=32, initial_weights=[15.0, -20.0], learning_rate=0.001)
```

I’d be very very thankful if you could fix any problem in here, so that fixing it can **make it such that** Adam **Optimizer here will work fine at lower learning rates (e.g. 0.002, 0.001…)**?