**Things to note:-**

- I prefer initialising weights at 0th epoch, and calculating loss at the end of each epoch, because that gives me the respective loss values of each weight (including the first), which ensures that I can plot the entire gradient descent process from first to last.
- During optimisation, I adjust gradient instead of learning rate, because Adam adjusts the learning rate differently in different directions, so it seems like dividing the gradient value makes sense because it has multiple components, while learning rate is a single scalar value.

The above two are my preferred design choices, and unless they’re causing error, please do not suggest me to change them. Instead fix the code while maintaining this design choice.

# Introduction

I'm **solving a simplified logical regression problem**, that only uses **weights w1 and w2** and **no bias term**. The **data used** here is **generated synthetically to nicely fit into such model**. (Not having a bias terms, reduces the parameters  to update during training to w1 and w2, making it possible for me to visualise the entire gradient descent process with only 3 dimension axes later on).

# Issue: doubtful about Adam optimiser’s behaviour

I am **highly doubtful that my implementation is correctly reflecting behaviour of the Adam** optimiser. In each step, the **weight updates seem to be very very small, and don’t seem to adaptively vary, regardless of the current position/epoch.** Also the **learning rate required here is much more than what Adam optimiser usually is initiated with.**

When setting learning rate 0.02 it takes around 130 epochs to converge (it will stay at low accuracy near 20%, until around 123th epoch, then make big jumps in the coming three epochs to reach near 80% accuracy).

**To my knowledge Adam optimiser is generally set to lower learning rates like 0.001**, and there onwards it automatically adjusts the learning rate to more reasonable rate. But **if I were to set my learning rate to 0.001 there, I’d need around 2536 epochs, just for a very simple logistic regression problem** with only two weights. 

Moreover, I observed that, the further you take the initial weights away from the minima, more learning rate/epochs will be needed, and there is no adaption of learning based on the current position, the weights updates are always small and steady regardless of where you start or which epoch you’re in.

I **feel like if Adam was correctly implemented** here, it **would have simply adjusted the learning rate during the training to converge much faster.** 

# Code

**Gradient Descent Class**

```python
class GradientDescent:
    def __init__(self, optimizer=None):
        self.weights = None
        self.learning_rate = None
        self.optimizer = optimizer

    def fit(self, X_train, y_train, X_val, y_val, epochs=100, initial_weights=None, learning_rate=0.002, batch_size=32, verbose=True):
        self.learning_rate = learning_rate

        # Initialize lists to store metrics for each epoch
        train_losses, train_accuracies, val_losses, val_accuracies, weight_history = [], [], [], [], []

        for epoch in range(epochs):
            if epoch == 0:
                np.random.seed(42)
                self.weights = np.array(initial_weights) if initial_weights else np.random.uniform(-1, 1, num_features)
            else:
                for i in range(0, X_train.shape[0], batch_size):
                    X_batch = X_train[i:i + batch_size]
                    y_true_batch = y_train[i:i + batch_size]
    
                    # Compute linear combination and apply sigmoid
                    linear_combination = np.dot(X_batch, self.weights)
                    y_pred_batch = 1 / (1 + np.exp(-linear_combination))
    
                    # Compute gradient of binary-crossentropy loss w.r.t weights 
                    grad_loss = np.dot(X_batch.T, (y_pred_batch - y_true_batch)) / X_batch.shape[0]
    
                    # Apply optimizer 
                    if self.optimizer:
                        grad_loss = self.optimizer.update(grad_loss)
    
                    # Update weights
                    self.weights -= self.learning_rate * grad_loss

            # Training evaluation after the epoch
            train_loss, train_accuracy = self.evaluate(X_train, y_train)
            val_loss, val_accuracy = self.evaluate(X_val, y_val)

            # Store metrics & display the metrics for the current epoch
            weight_history.append(self.weights.copy()); train_losses.append(train_loss); train_accuracies.append(train_accuracy); val_losses.append(val_loss); val_accuracies.append(val_accuracy)
            if verbose:
                print(f"Epoch {epoch + 1}:\nWeights: {self.weights}\nTrain Loss: {train_loss:.8f}, Train Accuracy: {train_accuracy:.4f}\nValidation Loss: {val_loss:.8f}, Validation Accuracy: {val_accuracy:.4f}\n")

        return train_losses, train_accuracies, val_losses, val_accuracies, weight_history

    def evaluate(self, X, y_true):
        linear_combination = np.dot(X, self.weights)
        y_pred = 1 / (1 + np.exp(-linear_combination))
        loss = -np.mean(y_true * np.log(y_pred + 1e-8) + (1 - y_true) * np.log(1 - y_pred + 1e-8)) # Cross-entropy loss
        accuracy = np.mean((y_pred >= 0.5) == y_true)
        return loss, accuracy
```

**Adam Optimizer class**

```python
class Adam:
    def __init__(self, beta1=0.9, beta2=0.999, epsilon=1e-7):
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.m_t = None  # First moment estimate (momentum)
        self.v_t = None  # Second moment estimate (RMSProp-like)
        self.t = 0       # Timestep for bias correction

    def update(self, gradient):
        if self.m_t is None:
            self.m_t = np.zeros_like(gradient)
        if self.v_t is None:
            self.v_t = np.zeros_like(gradient)

        self.t += 1

        # Update biased first and second moment estimates
        self.m_t = self.beta1 * self.m_t + (1 - self.beta1) * gradient
        self.v_t = self.beta2 * self.v_t + (1 - self.beta2) * (gradient ** 2)

        # Bias-corrected moment estimates
        m_hat = self.m_t / (1 - self.beta1 ** self.t)
        v_hat = self.v_t / (1 - self.beta2 ** self.t)

        # Update weights
        adjusted_gradient = m_hat / (np.sqrt(v_hat) + self.epsilon)

        return adjusted_gradient
```

**Performing Gradient Descent**

```python
# Generate Synthetic data
np.random.seed(42)
true_weights = np.array([-3, 3.5])
X = np.random.rand(500, len(true_weights)) * 10
y = (1 / (1 + np.exp(-np.dot(X, true_weights))) > 0.5).astype(int)
y[np.random.choice(500, int(0.1 * 500), replace=False)] ^= 1

# Split the data into training (60%), validation (20%), and test (20%) sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_test, y_test, test_size=0.5, random_state=42)

# Training
gd = GradientDescent(optimizer=Adam())

# very slowly converges at around 90th epoch (learning rate=0.03)
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=100, batch_size=32, initial_weights=[15.0, -20.0], learning_rate=0.03)

# very slowly converges at around 130th epoch (learning rate=0.02)
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=200, batch_size=32, initial_weights=[15.0, -20.0], learning_rate=0.02)

# very slowly converges at around 2540th epoch (learning rate=0.001)
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=3000, batch_size=32, initial_weights=[15.0, -20.0], learning_rate=0.001)

# you'll need 600 epochs at learning rate 0.02, if you take the initial weights further away to (-40, 10)
metrics = gd.fit(X_train, y_train, X_val, y_val, epochs=1000, batch_size=32, initial_weights=[-40.0, -20.0], learning_rate=0.02)
```

I’d be very very thankful if you could fix any problem in here, so that fixing it can **make it such that** Adam **optimizer here will work fine at lower learning rates (e.g. 0.002, 0.001…).** Or **make the weight update more adaptive depending on the current position/epoch.**