Repository: pyro-ppl/pyro

Issue:
Bayesian classifiers
We want to use some combination of bayesian regression and bayesian NNs (to be determined) as anchor models for the first pyro release. These can be implemented in a way that shares a lot of work, and makes it extensible. The approach is to define a parametrized family of classifiers as as a pytorch Module; usually one would do MLE on this family to train a classifier, instead we will define a "lift" operation that upgrades the parameters to random variables; we then do VI (or other inference) for the posterior over lifted parameters. See the extensive discussion of this approach in #40.

We should implement the lifting operation and then test it on Bayesian regression examples. Then we can try Bayesian NNs.


Discussion:
I can implement and test the `lift` operation, and someone else could do the example models.
i can do this (both or example) its in line with the other things im doing
For Bayesian regression, benchmark will be from the Stan ADVI paper: https://arxiv.org/abs/1506.03431
They simulate their own data for the regression example which isnt in the paper or repo. Going to try to get my hands on it otherwise use a different paper.
update: so finding a replicable benchmark is turning out to be a less trivial task than i foresaw.  essentially i need something that has all three things: data, model, results.

0) Stan paper: No data, contacted them for data (synthesized by them)
1) [drugowitsch's paper](https://arxiv.org/pdf/1310.5438.pdf) has data and model, but no real results. they run it on a toy example to show that it 'works'.
2) [edward](https://arxiv.org/pdf/1701.03757.pdf) has a regression example with a dataset i can access but dont report posterior results (they use it as a speed test comparison).
3) Gelman's book in Bayesian Data Analysis cites a dataset that i was able to get my hands on, but uses an improper distribution for alpha. After fiddling and discussing with @martinjankowiak, ill need to sample \alpha from truncated distribution to run VI on it.
4) [edit]  theres a hierarchical logistic regression in the Stan paper that uses the same data as above.

I will continue hunting for a published result that has all 3 of what i need, but i was thinking in the meantime of doing (2) by modifying their code to use klqp and comparing that with ours. It will give us confidence that we have something that is correct, though not officially in a paper. [edit:] for a published result i can do (4) and compare log predictives against theirs. @ngoodman thoughts?
never simple, eh? :)

i think it's acceptable for the performance benchmark to be an available system, rather than a result reported in a paper. so this would suggest taking a model+data example (or a couple) and comparing the log-predictive and run time to stan and/or edward. using simple examples from the book (eg rats) seems like a good idea?

(in some sense i am suggesting extending the scope of this "anchor model" to be comparison tests against stan and/or edward for a single model. it would be nice to put that together in such a way that it's easy to then use the pipeline to do that comparison for other models....)
status update:

running logistic regression against edward's model
- works with small generated data (they use)
- went back and forth for larger datasets (see discussion above), using UCI's Covtype 
  - works in pyro
  - WIP - trying to do this in edward/tf 
- compared by sampling posterior (because im not as familiar with tf/edward), will add log-predictive comparison
- did not do speedtest with small data will do with large
update:

- works with large dataset (up to a constant, could be improved with some tuning)
- will do speedtest on CPU (their reported numbers are on GPU or 12 CPU) 
- nan/learning errors using `random_module` + nn to define the prior rather than sampling directly. have both versions

Resolving patch:
diff --git a/examples/bayesian_regression.py b/examples/bayesian_regression.py
new file mode 100644
--- /dev/null
+++ b/examples/bayesian_regression.py
@@ -0,0 +1,151 @@
+import numpy as np
+import argparse
+import torch
+import torch.nn as nn
+from torch.nn.functional import normalize  # noqa: F401
+
+from torch.autograd import Variable
+
+import pyro
+from pyro import poutine
+from pyro.distributions import DiagNormal, Bernoulli  # noqa: F401
+from pyro.infer import SVI
+from pyro.optim import Adam
+
+"""
+Bayesian Regression
+"""
+
+
+# generate toy dataset
+def build_linear_dataset(N, noise_std=0.1):
+    D = 1
+    X = np.linspace(-6, 6, num=N)
+    y = 3 * X + 1 + np.random.normal(0, noise_std, size=N)
+    X = X.reshape((N, D))
+    y = y.reshape((N, 1))
+    X, y = Variable(torch.Tensor(X)), Variable(torch.Tensor(y))
+    return torch.cat((X, y), 1)
+
+
+def build_logistic_dataset(N, noise_std=0.1):
+    D = 1
+    X = np.linspace(-6, 6, num=N)
+    y = np.tanh(X) + np.random.normal(0, noise_std, size=N)
+    y[y < 0.5] = 0
+    y[y >= 0.5] = 1
+    X = (X - 4.0) / 4.0
+    X = X.reshape((N, D))
+    y = y.reshape((N, 1))
+    X, y = Variable(torch.Tensor(X)), Variable(torch.Tensor(y))
+    return torch.cat((X, y), 1)
+
+
+def log_reg(x_data):
+    p_w = pyro.param("weight", Variable(torch.zeros(D, 1), requires_grad=True))
+    p_b = pyro.param("bias", Variable(torch.ones(1), requires_grad=True))
+    reg = torch.addmm(p_b, x_data, p_w)
+    latent = sigmoid(reg)
+    return latent
+
+
+def lin_reg(x_data):
+    p_w = pyro.param("weight", Variable(torch.zeros(1), requires_grad=True))
+    p_b = pyro.param("bias", Variable(torch.ones(1), requires_grad=True))
+    latent = p_w.squeeze() * x_data.squeeze() + p_b
+    return latent
+
+
+sigmoid = nn.Sigmoid()
+softplus = nn.Softplus()
+
+N = 1000  # size of toy data
+D = 1  # number of features
+batch_size = 256  # batch size
+
+# type of regression
+reg_fn = lin_reg
+
+
+def model(data):
+    mu = Variable(torch.zeros(D, 1))
+    sigma = Variable(torch.ones(D, 1))
+    bias_mu = Variable(torch.zeros(1))
+    bias_sigma = Variable(torch.ones(1))
+    w_prior = DiagNormal(mu, sigma)
+    b_prior = DiagNormal(bias_mu, bias_sigma)
+    priors = {'weight': w_prior, 'bias': b_prior}
+    lifted_fn = poutine.lift(reg_fn, priors)
+
+    with pyro.iarange("map", N, subsample=data):
+        x_data = data[:, :-1]
+        y_data = data[:, -1]
+        latent = lifted_fn(x_data)
+        if reg_fn.__name__ == 'lin_reg':
+            pyro.observe("obs", DiagNormal(latent, Variable(torch.ones(data.size(0)))), y_data.squeeze())
+        else:
+            pyro.observe("obs", Bernoulli(latent.squeeze()), y_data)
+
+
+def guide(data):
+    x_data = data[:, :-1]
+    w_mu = Variable(torch.randn(D, 1), requires_grad=True)
+    w_log_sig = Variable(-3.0 * torch.ones(D, 1) + 0.05 * torch.randn(D, 1), requires_grad=True)
+    b_mu = Variable(torch.randn(1), requires_grad=True)
+    b_log_sig = Variable(-3.0 * torch.ones(1) + 0.05 * torch.randn(1), requires_grad=True)
+    mw_param = pyro.param("guide_mean_weight", w_mu)
+    sw_param = softplus(pyro.param("guide_sigma_weight", w_log_sig))
+    mb_param = pyro.param("guide_mean_bias", b_mu)
+    sb_param = softplus(pyro.param("guide_sigma_bias", b_log_sig))
+    w_prior = DiagNormal(mw_param, sw_param)
+    b_prior = DiagNormal(mb_param, sb_param)
+    priors = {'weight': w_prior, 'bias': b_prior}
+    with pyro.iarange("map", N, subsample=x_data):
+        lifted_fn = poutine.lift(reg_fn, priors)
+        lifted_fn(x_data)
+
+
+def load_data(reg_type):
+    global reg_fn
+    if reg_type == 'linear':
+        return build_linear_dataset(N)
+    elif reg_type == 'logistic':
+        reg_fn = log_reg
+        return build_logistic_dataset(N)
+    raise ValueError('data set type should be "logistic" or "linear".')
+
+
+adam = Adam({"lr": 0.01})
+svi = SVI(model, guide, adam, loss="ELBO")
+
+# get batch indices
+all_batches = np.arange(0, N, batch_size)
+if all_batches[-1] != N:
+    all_batches = list(all_batches) + [N]
+
+
+def main():
+    parser = argparse.ArgumentParser(description="parse args")
+    parser.add_argument('-n', '--num-epochs', default=1000, type=int)
+    parser.add_argument('-b', '--batch', default=False, type=bool)
+    parser.add_argument('-t', '--regression-type', default='linear', type=str)
+    args = parser.parse_args()
+    data = load_data(args.regression_type)
+    for j in range(args.num_epochs):
+        if not args.batch:
+            epoch_loss = svi.step(data)
+        else:
+            # mini batch
+            epoch_loss = 0.0
+            # shuffle data
+            data = data[torch.randperm(N)]
+            for ix, batch_start in enumerate(all_batches[:-1]):
+                batch_end = all_batches[ix + 1]
+                batch_data = data[batch_start: batch_end]
+                epoch_loss += svi.step(batch_data)
+        if j % 100 == 0:
+            print("epoch avg loss {}".format(epoch_loss/float(N)))
+
+
+if __name__ == '__main__':
+    main()


Test patch: