Repository: pytorch/pytorch
Pull request: Fixed distribution constraints and added some test cases for distributions parameter check
Hi,

This PR fixes an issue with the constraints: GreaterThan and LessThan. I have also added testcases to check that pytorch raises exceptions for invalid arguments for all distributions. 

Introduced a global flag `_validate_args` in `Distribution` which allows enabling `validate_args` for all distributions rather than enabling it for every distribution individually. Without this, adding this parameter for all distribution might be tedious for the user.

Also, fixed some existing issues with several distributions and tests.

This partially fixes [#5248](https://github.com/pytorch/pytorch/issues/5248) and can go along with [#133](https://github.com/probtorch/pytorch/pull/133)

This has been reviewed in [probtorch#134](https://github.com/probtorch/pytorch/pull/134)

Problem statement:
Pytorch inconsistent behavior for boundary checks
Hi,

I observe inconsistent behavior when running the Bernoulli distribution with invalid values (code snippets below). When I run it with input that is outside the expected range (`2.0` in this case) the `sample` function fails as expected, with this message:

```bash
python /home/torchtest.py
Traceback (most recent call last):
  File "/home/torchtest.py", line 9, in <module>
    print(b.sample())
  File "/home/pytorch/torch/distributions/bernoulli.py", line 75, in sample
    return torch.bernoulli(self.probs.expand(shape))
RuntimeError: invalid argument 1: must be >= 0 and <= 1 at /home/pytorch/aten/src/TH/THRandom.cpp:343
```

Code:
```Python
p = torch.Tensor([2.0])
b = torch.distributions.Bernoulli(p)
print(b.sample())
```

but the `log_prob` function does not fail and returns the following:
```bash
-15.9424
[torch.FloatTensor of size 1]
```

Code:
```Python
p = torch.Tensor([2.0])
b = torch.distributions.Bernoulli(p)
print(b.log_prob((torch.Tensor([0]))))
```
Is this expected? Shouldn't both functions (`sample` and `log_prob`) fail with the same error?

Hints from issue discussion:
cc: @fritzo @neerajprad 
I don't think we've considered this scenario during our meetings
We are not validating arguments provided to the distribution constructors, as far as I know. The `sample` call fails because `torch.bernoulli` internally does a validation check of the arguments, which `log_prob` does not need to. To prevent such differences in behavior, we could add a validation check for distribution parameters in the constructor for all distributions. If there is a significant performance impact, we could also support a mode that skips these checks. What do you think?
I have mixed feelings. On one hand we're ok with paying the performance penalty in `log_prob`, on the other hand, putting those checks everywhere seems kinda expensive.
I think the `.__init__()` method would be the right place for these checks if we decide to perform them always, but I worry that this would cause significant performance overhead, especially since we consider distributions to be flyweight objects that merely wrap methods. Maybe we could add a new method `.validate()` to do this expensive validation logic?
I also think that the right place for these validation checks should be the constructor, so that any penalty is paid once and not multiple times as we call `sample` or `log_prob` on the same instance. This also ensures that we fail fast, and prevents ad-hoc behavior like this that results from shifting the onus of argument validation to the underlying sampler. Adding a `validate=False` argument to the constructor (or a separate `.validate()` method) might be a reasonable step forward, given that performance impact could be non trivial for large tensors.
Like @neerajprad mentioned, I think it would be useful to have an optional debug flag which, when set, would cause pytorch to check for boundary conditions for all distributions instead of having checks for individual distributions. Tensorflow has a similar flag, [validate_args](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/distributions/Bernoulli#validate_args), which is an optional argument for all distributions. Also, it seems like torch already has some of these checks in place for some distributions(may not be all). I would be happy to contribute the feature and the test cases, if this proposal sounds reasonable. Any suggestions on how to proceed?
@lazypanda1 Sure, I think an argument `validate_args=False` make sense. You can even add a generic test to tests/test_distributions.py
```py
class TestValidation(TestCase):
    def test_valid(self):
        for Dist, params in EXAMPLES:
            for i, param in enumerate(params):
                Dist(validate_args=True, **param)  # should not error

    def test_invalid(self):
        # this requires actual bad examples
```
This will ensure that all distributions take a `validate_args` argument. We may also want to add a `validate_args` argument to `Transform`s in a later PR.
Should I print the tensor as well when the values are invalid?
No, printing whole tensors is rarely a good idea. It takes up too much space
@lazypanda1 It looks like @vishwakftw just started to add some validation logic in https://github.com/probtorch/pytorch/pull/133. If you'd like early code review on your PR, feel free to send a work-in-progress PR to probtorch/pytorch. We've been doing that for alot of the distributions PRs: first a review on probtorch/pytorch, then close that and open a final PR for review on pytorch/pytorch.

Commit message: Replaced Variable with variable

Patch to review:
diff --git a/test/test_distributions.py b/test/test_distributions.py
index c2661811ed9d..69a990157575 100644
--- a/test/test_distributions.py
+++ b/test/test_distributions.py
@@ -31,7 +31,7 @@
 
 import torch
 from common import TestCase, run_tests, set_rng_seed
-from torch.autograd import Variable, grad, gradcheck
+from torch.autograd import Variable, grad, gradcheck, variable
 from torch.distributions import (Bernoulli, Beta, Binomial, Categorical,
                                  Cauchy, Chi2, Dirichlet, Distribution,
                                  Exponential, ExponentialFamily,
@@ -86,138 +86,138 @@ def is_all_nan(tensor):
 Example = namedtuple('Example', ['Dist', 'params'])
 EXAMPLES = [
     Example(Bernoulli, [
-        {'probs': Variable(torch.Tensor([0.7, 0.2, 0.4]), requires_grad=True)},
-        {'probs': Variable(torch.Tensor([0.3]), requires_grad=True)},
+        {'probs': variable([0.7, 0.2, 0.4], requires_grad=True)},
+        {'probs': variable([0.3], requires_grad=True)},
         {'probs': 0.3},
     ]),
     Example(Geometric, [
-        {'probs': Variable(torch.Tensor([0.7, 0.2, 0.4]), requires_grad=True)},
-        {'probs': Variable(torch.Tensor([0.3]), requires_grad=True)},
+        {'probs': variable([0.7, 0.2, 0.4], requires_grad=True)},
+        {'probs': variable([0.3], requires_grad=True)},
         {'probs': 0.3},
     ]),
     Example(Beta, [
         {
-            'concentration1': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
-            'concentration0': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
+            'concentration1': variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
+            'concentration0': variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
         },
         {
-            'concentration1': Variable(torch.exp(torch.randn(4)), requires_grad=True),
-            'concentration0': Variable(torch.exp(torch.randn(4)), requires_grad=True),
+            'concentration1': variable(torch.exp(torch.randn(4)), requires_grad=True),
+            'concentration0': variable(torch.exp(torch.randn(4)), requires_grad=True),
         },
     ]),
     Example(Categorical, [
-        {'probs': Variable(torch.Tensor([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]), requires_grad=True)},
-        {'probs': Variable(torch.Tensor([[1.0, 0.0], [0.0, 1.0]]), requires_grad=True)},
+        {'probs': variable([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]], requires_grad=True)},
+        {'probs': variable([[1.0, 0.0], [0.0, 1.0]], requires_grad=True)},
     ]),
     Example(Binomial, [
-        {'probs': Variable(torch.Tensor([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]), requires_grad=True), 'total_count': 10},
-        {'probs': Variable(torch.Tensor([[1.0, 0.0], [0.0, 1.0]]), requires_grad=True), 'total_count': 10},
+        {'probs': variable([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]], requires_grad=True), 'total_count': 10},
+        {'probs': variable([[1.0, 0.0], [0.0, 1.0]], requires_grad=True), 'total_count': 10},
     ]),
     Example(Multinomial, [
-        {'probs': Variable(torch.Tensor([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]), requires_grad=True), 'total_count': 10},
-        {'probs': Variable(torch.Tensor([[1.0, 0.0], [0.0, 1.0]]), requires_grad=True), 'total_count': 10},
+        {'probs': variable([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]], requires_grad=True), 'total_count': 10},
+        {'probs': variable([[1.0, 0.0], [0.0, 1.0]], requires_grad=True), 'total_count': 10},
     ]),
     Example(Cauchy, [
         {'loc': 0.0, 'scale': 1.0},
-        {'loc': Variable(torch.Tensor([0.0])), 'scale': 1.0},
-        {'loc': Variable(torch.Tensor([[0.0], [0.0]])),
-         'scale': Variable(torch.Tensor([[1.0], [1.0]]))}
+        {'loc': variable([0.0]), 'scale': 1.0},
+        {'loc': variable([[0.0], [0.0]]),
+         'scale': variable([[1.0], [1.0]])}
     ]),
     Example(Chi2, [
-        {'df': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)},
-        {'df': Variable(torch.exp(torch.randn(1)), requires_grad=True)},
+        {'df': variable(torch.exp(torch.randn(2, 3)), requires_grad=True)},
+        {'df': variable(torch.exp(torch.randn(1)), requires_grad=True)},
     ]),
     Example(StudentT, [
-        {'df': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)},
-        {'df': Variable(torch.exp(torch.randn(1)), requires_grad=True)},
+        {'df': variable(torch.exp(torch.randn(2, 3)), requires_grad=True)},
+        {'df': variable(torch.exp(torch.randn(1)), requires_grad=True)},
     ]),
     Example(Dirichlet, [
-        {'concentration': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)},
-        {'concentration': Variable(torch.exp(torch.randn(4)), requires_grad=True)},
+        {'concentration': variable(torch.exp(torch.randn(2, 3)), requires_grad=True)},
+        {'concentration': variable(torch.exp(torch.randn(4)), requires_grad=True)},
     ]),
     Example(Exponential, [
-        {'rate': Variable(torch.randn(5, 5).abs(), requires_grad=True)},
-        {'rate': Variable(torch.randn(1).abs(), requires_grad=True)},
+        {'rate': variable(torch.randn(5, 5).abs(), requires_grad=True)},
+        {'rate': variable(torch.randn(1).abs(), requires_grad=True)},
     ]),
     Example(FisherSnedecor, [
         {
-            'df1': Variable(torch.randn(5, 5).abs(), requires_grad=True),
-            'df2': Variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'df1': variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'df2': variable(torch.randn(5, 5).abs(), requires_grad=True),
         },
         {
-            'df1': Variable(torch.randn(1).abs(), requires_grad=True),
-            'df2': Variable(torch.randn(1).abs(), requires_grad=True),
+            'df1': variable(torch.randn(1).abs(), requires_grad=True),
+            'df2': variable(torch.randn(1).abs(), requires_grad=True),
         },
         {
-            'df1': Variable(torch.Tensor([1.0])),
+            'df1': variable([1.0]),
             'df2': 1.0,
         }
     ]),
     Example(Gamma, [
         {
-            'concentration': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
-            'rate': Variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
+            'concentration': variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
+            'rate': variable(torch.exp(torch.randn(2, 3)), requires_grad=True),
         },
         {
-            'concentration': Variable(torch.exp(torch.randn(1)), requires_grad=True),
-            'rate': Variable(torch.exp(torch.randn(1)), requires_grad=True),
+            'concentration': variable(torch.exp(torch.randn(1)), requires_grad=True),
+            'rate': variable(torch.exp(torch.randn(1)), requires_grad=True),
         },
     ]),
     Example(Gumbel, [
         {
-            'loc': Variable(torch.randn(5, 5), requires_grad=True),
-            'scale': Variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'loc': variable(torch.randn(5, 5), requires_grad=True),
+            'scale': variable(torch.randn(5, 5).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.randn(1), requires_grad=True),
-            'scale': Variable(torch.randn(1).abs(), requires_grad=True),
+            'loc': variable(torch.randn(1), requires_grad=True),
+            'scale': variable(torch.randn(1).abs(), requires_grad=True),
         },
     ]),
     Example(Laplace, [
         {
-            'loc': Variable(torch.randn(5, 5), requires_grad=True),
-            'scale': Variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'loc': variable(torch.randn(5, 5), requires_grad=True),
+            'scale': variable(torch.randn(5, 5).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.randn(1), requires_grad=True),
-            'scale': Variable(torch.randn(1).abs(), requires_grad=True),
+            'loc': variable(torch.randn(1), requires_grad=True),
+            'scale': variable(torch.randn(1).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.Tensor([1.0, 0.0]), requires_grad=True),
-            'scale': Variable(torch.Tensor([1e-5, 1e-5]), requires_grad=True),
+            'loc': variable([1.0, 0.0], requires_grad=True),
+            'scale': variable([1e-5, 1e-5], requires_grad=True),
         },
     ]),
     Example(LogNormal, [
         {
-            'loc': Variable(torch.randn(5, 5), requires_grad=True),
-            'scale': Variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'loc': variable(torch.randn(5, 5), requires_grad=True),
+            'scale': variable(torch.randn(5, 5).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.randn(1), requires_grad=True),
-            'scale': Variable(torch.randn(1).abs(), requires_grad=True),
+            'loc': variable(torch.randn(1), requires_grad=True),
+            'scale': variable(torch.randn(1).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.Tensor([1.0, 0.0]), requires_grad=True),
-            'scale': Variable(torch.Tensor([1e-5, 1e-5]), requires_grad=True),
+            'loc': variable([1.0, 0.0], requires_grad=True),
+            'scale': variable([1e-5, 1e-5], requires_grad=True),
         },
     ]),
     Example(Normal, [
         {
-            'loc': Variable(torch.randn(5, 5), requires_grad=True),
-            'scale': Variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'loc': variable(torch.randn(5, 5), requires_grad=True),
+            'scale': variable(torch.randn(5, 5).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.randn(1), requires_grad=True),
-            'scale': Variable(torch.randn(1).abs(), requires_grad=True),
+            'loc': variable(torch.randn(1), requires_grad=True),
+            'scale': variable(torch.randn(1).abs(), requires_grad=True),
         },
         {
-            'loc': Variable(torch.Tensor([1.0, 0.0]), requires_grad=True),
-            'scale': Variable(torch.Tensor([1e-5, 1e-5]), requires_grad=True),
+            'loc': variable([1.0, 0.0], requires_grad=True),
+            'scale': variable([1e-5, 1e-5], requires_grad=True),
         },
     ]),
     Example(OneHotCategorical, [
-        {'probs': Variable(torch.Tensor([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]), requires_grad=True)},
-        {'probs': Variable(torch.Tensor([[1.0, 0.0], [0.0, 1.0]]), requires_grad=True)},
+        {'probs': variable([[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]], requires_grad=True)},
+        {'probs': variable([[1.0, 0.0], [0.0, 1.0]], requires_grad=True)},
     ]),
     Example(Pareto, [
         {
@@ -225,20 +225,20 @@ def is_all_nan(tensor):
             'alpha': 1.0
         },
         {
-            'scale': Variable(torch.randn(5, 5).abs(), requires_grad=True),
-            'alpha': Variable(torch.randn(5, 5).abs(), requires_grad=True)
+            'scale': variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'alpha': variable(torch.randn(5, 5).abs(), requires_grad=True)
         },
         {
-            'scale': torch.tensor([1.0]),
+            'scale': variable([1.0]),
             'alpha': 1.0
         }
     ]),
     Example(Poisson, [
         {
-            'rate': Variable(torch.randn(5, 5).abs(), requires_grad=True),
+            'rate': variable(torch.randn(5, 5).abs(), requires_grad=True),
         },
         {
-            'rate': Variable(torch.randn(3).abs(), requires_grad=True),
+            'rate': variable(torch.randn(3).abs(), requires_grad=True),
         },
         {
             'rate': 0.2,
@@ -246,66 +246,245 @@ def is_all_nan(tensor):
     ]),
     Example(RelaxedBernoulli, [
         {
-            'temperature': Variable(torch.Tensor([0.5]), requires_grad=True),
-            'probs': Variable(torch.Tensor([0.7, 0.2, 0.4]), requires_grad=True),
+            'temperature': variable([0.5], requires_grad=True),
+            'probs': variable([0.7, 0.2, 0.4], requires_grad=True),
         },
         {
-            'temperature': Variable(torch.Tensor([2.0])),
-            'probs': Variable(torch.Tensor([0.3])),
+            'temperature': variable([2.0]),
+            'probs': variable([0.3]),
         },
         {
-            'temperature': Variable(torch.Tensor([7.2])),
-            'logits': Variable(torch.Tensor([-2.0, 2.0, 1.0, 5.0]))
+            'temperature': variable([7.2]),
+            'logits': variable([-2.0, 2.0, 1.0, 5.0])
         }
     ]),
     Example(RelaxedOneHotCategorical, [
         {
-            'temperature': Variable(torch.Tensor([0.5]), requires_grad=True),
-            'probs': Variable(torch.Tensor([[0.1, 0.2, 0.7], [0.5, 0.3, 0.2]]), requires_grad=True)
+            'temperature': variable([0.5], requires_grad=True),
+            'probs': variable([[0.1, 0.2, 0.7], [0.5, 0.3, 0.2]], requires_grad=True)
         },
         {
-            'temperature': Variable(torch.Tensor([2.0])),
-            'probs': Variable(torch.Tensor([[1.0, 0.0], [0.0, 1.0]]))
+            'temperature': variable([2.0]),
+            'probs': variable([[1.0, 0.0], [0.0, 1.0]])
         },
         {
-            'temperature': Variable(torch.Tensor([7.2])),
-            'logits': Variable(torch.Tensor([[-2.0, 2.0], [1.0, 5.0]]))
+            'temperature': variable([7.2]),
+            'logits': variable([[-2.0, 2.0], [1.0, 5.0]])
         }
     ]),
     Example(TransformedDistribution, [
         {
-            'base_distribution': Normal(Variable(torch.randn(2, 3), requires_grad=True),
-                                        Variable(torch.randn(2, 3).abs(), requires_grad=True)),
+            'base_distribution': Normal(variable(torch.randn(2, 3), requires_grad=True),
+                                        variable(torch.randn(2, 3).abs(), requires_grad=True)),
             'transforms': [],
         },
         {
-            'base_distribution': Normal(Variable(torch.randn(2, 3), requires_grad=True),
-                                        Variable(torch.randn(2, 3).abs(), requires_grad=True)),
+            'base_distribution': Normal(variable(torch.randn(2, 3), requires_grad=True),
+                                        variable(torch.randn(2, 3).abs(), requires_grad=True)),
             'transforms': ExpTransform(),
         },
         {
-            'base_distribution': Normal(Variable(torch.randn(2, 3, 5), requires_grad=True),
-                                        Variable(torch.randn(2, 3, 5).abs(), requires_grad=True)),
-            'transforms': [AffineTransform(Variable(torch.randn(3, 5)), Variable(torch.randn(3, 5))),
+            'base_distribution': Normal(variable(torch.randn(2, 3, 5), requires_grad=True),
+                                        variable(torch.randn(2, 3, 5).abs(), requires_grad=True)),
+            'transforms': [AffineTransform(variable(torch.randn(3, 5)), variable(torch.randn(3, 5))),
                            ExpTransform()],
         },
     ]),
     Example(Uniform, [
         {
-            'low': Variable(torch.zeros(5, 5), requires_grad=True),
-            'high': Variable(torch.ones(5, 5), requires_grad=True),
+            'low': variable(torch.zeros(5, 5), requires_grad=True),
+            'high': variable(torch.ones(5, 5), requires_grad=True),
         },
         {
-            'low': Variable(torch.zeros(1), requires_grad=True),
-            'high': Variable(torch.ones(1), requires_grad=True),
+            'low': variable(torch.zeros(1), requires_grad=True),
+            'high': variable(torch.ones(1), requires_grad=True),
         },
         {
-            'low': Variable(torch.Tensor([1.0, 1.0]), requires_grad=True),
-            'high': Variable(torch.Tensor([2.0, 3.0]), requires_grad=True),
+            'low': variable([1.0, 1.0], requires_grad=True),
+            'high': variable([2.0, 3.0], requires_grad=True),
         },
     ]),
 ]
 
+BAD_EXAMPLES = [
+    Example(Bernoulli, [
+        {'probs': variable([1.1, 0.2, 0.4], requires_grad=True)},
+        {'probs': variable([-0.5], requires_grad=True)},
+        {'probs': 1.00001},
+    ]),
+    Example(Beta, [
+        {
+            'concentration1': variable([0.0], requires_grad=True),
+            'concentration0': variable([0.0], requires_grad=True),
+        },
+        {
+            'concentration1': variable([-1.0], requires_grad=True),
+            'concentration0': variable([-2.0], requires_grad=True),
+        },
+    ]),
+    Example(Geometric, [
+        {'probs': variable([1.1, 0.2, 0.4], requires_grad=True)},
+        {'probs': variable([-0.3], requires_grad=True)},
+        {'probs': 1.00000001},
+    ]),
+    Example(Categorical, [
+        {'probs': variable([[-0.1, 0.2, 0.3], [0.5, 0.3, 0.2]], requires_grad=True)},
+        {'probs': variable([[-1.0, 10.0], [0.0, -1.0]], requires_grad=True)},
+    ]),
+    Example(Binomial, [
+        {'probs': variable([[-0.0000001, 0.2, 0.3], [0.5, 0.3, 0.2]], requires_grad=True),
+         'total_count': 10},
+        {'probs': variable([[1.0, 0.0], [0.0, 2.0]], requires_grad=True),
+         'total_count': 10},
+    ]),
+    Example(Cauchy, [
+        {'loc': 0.0, 'scale': -1.0},
+        {'loc': variable([0.0]), 'scale': 0.0},
+        {'loc': variable([[0.0], [-2.0]]),
+         'scale': variable([[-0.000001], [1.0]])}
+    ]),
+    Example(Chi2, [
+        {'df': variable([0], requires_grad=True)},
+        {'df': variable([-2], requires_grad=True)},
+    ]),
+    Example(StudentT, [
+        {'df': variable([0], requires_grad=True)},
+        {'df': variable([-2], requires_grad=True)},
+    ]),
+    Example(Dirichlet, [
+        {'concentration': variable([0], requires_grad=True)},
+        {'concentration': variable([-2], requires_grad=True)}
+    ]),
+    Example(Exponential, [
+        {'rate': variable([0, 0], requires_grad=True)},
+        {'rate': variable([-2], requires_grad=True)}
+    ]),
+    Example(FisherSnedecor, [
+        {
+            'df1': variable([0, 0], requires_grad=True),
+            'df2': variable([-1, -100], requires_grad=True),
+        },
+        {
+            'df1': variable([1, 1], requires_grad=True),
+            'df2': variable([0, 0], requires_grad=True),
+        }
+    ]),
+    Example(Gamma, [
+        {
+            'concentration': variable([0, 0], requires_grad=True),
+            'rate': variable([-1, -100], requires_grad=True),
+        },
+        {
+            'concentration': variable([1, 1], requires_grad=True),
+            'rate': variable([0, 0], requires_grad=True),
+        }
+    ]),
+    Example(Gumbel, [
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([0, 1], requires_grad=True),
+        },
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([1, -1], requires_grad=True),
+        },
+    ]),
+    Example(Laplace, [
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([0, 1], requires_grad=True),
+        },
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([1, -1], requires_grad=True),
+        },
+    ]),
+    Example(LogNormal, [
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([0, 1], requires_grad=True),
+        },
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([1, -1], requires_grad=True),
+        },
+    ]),
+    Example(Normal, [
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([0, 1], requires_grad=True),
+        },
+        {
+            'loc': variable([1, 1], requires_grad=True),
+            'scale': variable([1, -1], requires_grad=True),
+        },
+        {
+            'loc': variable([1.0, 0.0], requires_grad=True),
+            'scale': variable([1e-5, -1e-5], requires_grad=True),
+        },
+    ]),
+    Example(OneHotCategorical, [
+        {'probs': variable([[0.1, 0.2, 0.3], [0.1, -10.0, 0.2]], requires_grad=True)},
+        {'probs': variable([[0.0, 0.0], [0.0, 0.0]], requires_grad=True)},
+    ]),
+    Example(Pareto, [
+        {
+            'scale': 0.0,
+            'alpha': 0.0
+        },
+        {
+            'scale': variable([0.0, 0.0], requires_grad=True),
+            'alpha': variable([-1e-5, 0.0], requires_grad=True)
+        },
+        {
+            'scale': variable([1.0]),
+            'alpha': -1.0
+        }
+    ]),
+    Example(Poisson, [
+        {
+            'rate': variable([0.0], requires_grad=True),
+        },
+        {
+            'rate': -1.0,
+        }
+    ]),
+    Example(RelaxedBernoulli, [
+        {
+            'temperature': variable([1.5], requires_grad=True),
+            'probs': variable([1.7, 0.2, 0.4], requires_grad=True),
+        },
+        {
+            'temperature': variable([2.0]),
+            'probs': variable([-1.0]),
+        }
+    ]),
+    Example(RelaxedOneHotCategorical, [
+        {
+            'temperature': variable([0.5], requires_grad=True),
+            'probs': variable([[-0.1, 0.2, 0.7], [0.5, 0.3, 0.2]], requires_grad=True)
+        },
+        {
+            'temperature': variable([2.0]),
+            'probs': variable([[-1.0, 0.0], [-1.0, 1.1]])
+        }
+    ]),
+    Example(Uniform, [
+        {
+            'low': variable([2.0], requires_grad=True),
+            'high': variable([2.0], requires_grad=True),
+        },
+        {
+            'low': variable([0.0], requires_grad=True),
+            'high': variable([0.0], requires_grad=True),
+        },
+        {
+            'low': variable([1.0], requires_grad=True),
+            'high': variable([0.0], requires_grad=True),
+        }
+    ])
+]
+
 
 def unwrap(value):
     if isinstance(value, Variable):
@@ -385,8 +564,8 @@ def _check_enumerate_support(self, dist, examples):
             expected = torch.Tensor(expected)
             actual = dist(param).enumerate_support()
             self.assertEqual(actual, expected)
-            param = Variable(param)
-            expected = Variable(expected)
+            param = variable(param)
+            expected = variable(expected)
             actual = dist(param).enumerate_support()
             self.assertEqual(actual, expected)
 
@@ -468,15 +647,15 @@ def test_bernoulli_enumerate_support(self):
         self._check_enumerate_support(Bernoulli, examples)
 
     def test_bernoulli_3d(self):
-        p = Variable(torch.Tensor(2, 3, 5).fill_(0.5), requires_grad=True)
+        p = variable(torch.Tensor(2, 3, 5).fill_(0.5), requires_grad=True)
         self.assertEqual(Bernoulli(p).sample().size(), (2, 3, 5))
         self.assertEqual(Bernoulli(p).sample(sample_shape=(2, 5)).size(),
                          (2, 5, 2, 3, 5))
         self.assertEqual(Bernoulli(p).sample((2,)).size(), (2, 2, 3, 5))
 
     def test_geometric(self):
-        p = torch.tensor([0.7, 0.2, 0.4], requires_grad=True)
-        r = torch.tensor(0.3, requires_grad=True)
+        p = variable([0.7, 0.2, 0.4], requires_grad=True)
+        r = variable(0.3, requires_grad=True)
         s = 0.3
         self.assertEqual(Geometric(p).sample((8,)).size(), (8, 3))
         self.assertEqual(Geometric(1).sample(), 0)
@@ -493,7 +672,7 @@ def test_geometric(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_geometric_log_prob_and_entropy(self):
-        p = Variable(torch.Tensor([0.7, 0.2, 0.4]), requires_grad=True)
+        p = variable([0.7, 0.2, 0.4], requires_grad=True)
         s = 0.3
 
         def ref_log_prob(idx, val, log_prob):
@@ -516,7 +695,7 @@ def test_geometric_sample(self):
                                          'Geometric(prob={})'.format(prob))
 
     def test_binomial(self):
-        p = Variable(torch.arange(0.05, 1, 0.1), requires_grad=True)
+        p = variable(torch.arange(0.05, 1, 0.1), requires_grad=True)
         for total_count in [1, 2, 10]:
             self._gradcheck_log_prob(lambda p: Binomial(total_count, p), [p])
             self._gradcheck_log_prob(lambda p: Binomial(total_count, None, p.log()), [p])
@@ -525,7 +704,7 @@ def test_binomial(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_binomial_log_prob(self):
-        probs = Variable(torch.arange(0.05, 1, 0.1))
+        probs = variable(torch.arange(0.05, 1, 0.1))
         for total_count in [1, 2, 10]:
 
             def ref_log_prob(idx, x, log_prob):
@@ -550,7 +729,7 @@ def test_binomial_extreme_vals(self):
 
     def test_multinomial_1d(self):
         total_count = 10
-        p = Variable(torch.Tensor([0.1, 0.2, 0.3]), requires_grad=True)
+        p = variable([0.1, 0.2, 0.3], requires_grad=True)
         self.assertEqual(Multinomial(total_count, p).sample().size(), (3,))
         self.assertEqual(Multinomial(total_count, p).sample((2, 2)).size(), (2, 2, 3))
         self.assertEqual(Multinomial(total_count, p).sample((1,)).size(), (1, 3))
@@ -561,7 +740,7 @@ def test_multinomial_1d(self):
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_multinomial_1d_log_prob(self):
         total_count = 10
-        p = Variable(torch.Tensor([0.1, 0.2, 0.3]), requires_grad=True)
+        p = variable([0.1, 0.2, 0.3], requires_grad=True)
         dist = Multinomial(total_count, probs=p)
         x = dist.sample()
         log_prob = dist.log_prob(x)
@@ -578,8 +757,8 @@ def test_multinomial_2d(self):
         total_count = 10
         probabilities = [[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]
         probabilities_1 = [[1.0, 0.0], [0.0, 1.0]]
-        p = Variable(torch.Tensor(probabilities), requires_grad=True)
-        s = Variable(torch.Tensor(probabilities_1), requires_grad=True)
+        p = variable(probabilities, requires_grad=True)
+        s = variable(probabilities_1, requires_grad=True)
         self.assertEqual(Multinomial(total_count, p).sample().size(), (2, 3))
         self.assertEqual(Multinomial(total_count, p).sample(sample_shape=(3, 4)).size(), (3, 4, 2, 3))
         self.assertEqual(Multinomial(total_count, p).sample((6,)).size(), (6, 2, 3))
@@ -596,7 +775,7 @@ def test_multinomial_2d(self):
         self.assertRaises(NotImplementedError, Multinomial(10, p).entropy)
 
     def test_categorical_1d(self):
-        p = Variable(torch.Tensor([0.1, 0.2, 0.3]), requires_grad=True)
+        p = variable([0.1, 0.2, 0.3], requires_grad=True)
         self.assertTrue(is_all_nan(Categorical(p).mean))
         self.assertTrue(is_all_nan(Categorical(p).variance))
         self.assertEqual(Categorical(p).sample().size(), ())
@@ -609,8 +788,8 @@ def test_categorical_1d(self):
     def test_categorical_2d(self):
         probabilities = [[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]
         probabilities_1 = [[1.0, 0.0], [0.0, 1.0]]
-        p = Variable(torch.Tensor(probabilities), requires_grad=True)
-        s = Variable(torch.Tensor(probabilities_1), requires_grad=True)
+        p = variable(probabilities, requires_grad=True)
+        s = variable(probabilities_1, requires_grad=True)
         self.assertEqual(Categorical(p).mean.size(), (2,))
         self.assertEqual(Categorical(p).variance.size(), (2,))
         self.assertTrue(is_all_nan(Categorical(p).mean))
@@ -644,7 +823,7 @@ def test_categorical_enumerate_support(self):
         self._check_enumerate_support(Categorical, examples)
 
     def test_one_hot_categorical_1d(self):
-        p = Variable(torch.Tensor([0.1, 0.2, 0.3]), requires_grad=True)
+        p = variable([0.1, 0.2, 0.3], requires_grad=True)
         self.assertEqual(OneHotCategorical(p).sample().size(), (3,))
         self.assertTrue(isinstance(OneHotCategorical(p).sample().data, torch.Tensor))
         self.assertEqual(OneHotCategorical(p).sample((2, 2)).size(), (2, 2, 3))
@@ -655,8 +834,8 @@ def test_one_hot_categorical_1d(self):
     def test_one_hot_categorical_2d(self):
         probabilities = [[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]
         probabilities_1 = [[1.0, 0.0], [0.0, 1.0]]
-        p = Variable(torch.Tensor(probabilities), requires_grad=True)
-        s = Variable(torch.Tensor(probabilities_1), requires_grad=True)
+        p = variable(probabilities, requires_grad=True)
+        s = variable(probabilities_1, requires_grad=True)
         self.assertEqual(OneHotCategorical(p).sample().size(), (2, 3))
         self.assertEqual(OneHotCategorical(p).sample(sample_shape=(3, 4)).size(), (3, 4, 2, 3))
         self.assertEqual(OneHotCategorical(p).sample((6,)).size(), (6, 2, 3))
@@ -674,8 +853,8 @@ def test_one_hot_categorical_enumerate_support(self):
         self._check_enumerate_support(OneHotCategorical, examples)
 
     def test_poisson_shape(self):
-        rate = Variable(torch.randn(2, 3).abs(), requires_grad=True)
-        rate_1d = Variable(torch.randn(1).abs(), requires_grad=True)
+        rate = variable(torch.randn(2, 3).abs(), requires_grad=True)
+        rate_1d = variable(torch.randn(1).abs(), requires_grad=True)
         self.assertEqual(Poisson(rate).sample().size(), (2, 3))
         self.assertEqual(Poisson(rate).sample((7,)).size(), (7, 2, 3))
         self.assertEqual(Poisson(rate_1d).sample().size(), (1,))
@@ -684,8 +863,8 @@ def test_poisson_shape(self):
 
     @unittest.skipIf(not TEST_NUMPY, "Numpy not found")
     def test_poisson_log_prob(self):
-        rate = Variable(torch.randn(2, 3).abs(), requires_grad=True)
-        rate_1d = Variable(torch.randn(1).abs(), requires_grad=True)
+        rate = variable(torch.randn(2, 3).abs(), requires_grad=True)
+        rate_1d = variable(torch.randn(1).abs(), requires_grad=True)
 
         def ref_log_prob(idx, x, log_prob):
             l = rate.data.view(-1)[idx]
@@ -758,7 +937,7 @@ def sample(self, *args, **kwargs):
             self.assertEqual(equal_probs, s)
 
     def test_relaxed_one_hot_categorical_1d(self):
-        p = Variable(torch.Tensor([0.1, 0.2, 0.3]), requires_grad=True)
+        p = variable([0.1, 0.2, 0.3], requires_grad=True)
         temp = torch.tensor(0.67, requires_grad=True)
         self.assertEqual(RelaxedOneHotCategorical(probs=p, temperature=temp).sample().size(), (3,))
         self.assertTrue(isinstance(RelaxedOneHotCategorical(probs=p, temperature=temp).sample().data, torch.Tensor))
@@ -769,10 +948,10 @@ def test_relaxed_one_hot_categorical_1d(self):
     def test_relaxed_one_hot_categorical_2d(self):
         probabilities = [[0.1, 0.2, 0.3], [0.5, 0.3, 0.2]]
         probabilities_1 = [[1.0, 0.0], [0.0, 1.0]]
-        temp = Variable(torch.Tensor([3.00]), requires_grad=True)
-        temp_2 = Variable(torch.Tensor([0.2]), requires_grad=True)
-        p = Variable(torch.Tensor(probabilities), requires_grad=True)
-        s = Variable(torch.Tensor(probabilities_1), requires_grad=True)
+        temp = variable([3.00], requires_grad=True)
+        temp_2 = variable([0.2], requires_grad=True)
+        p = variable(probabilities, requires_grad=True)
+        s = variable(probabilities_1, requires_grad=True)
         self.assertEqual(RelaxedOneHotCategorical(temp, p).sample().size(), (2, 3))
         self.assertEqual(RelaxedOneHotCategorical(temp, p).sample(sample_shape=(3, 4)).size(), (3, 4, 2, 3))
         self.assertEqual(RelaxedOneHotCategorical(temp, p).sample_n(6).size(), (6, 2, 3))
@@ -814,10 +993,10 @@ def pmf(self, samples):
             self.assertEqual(equal_probs, s)
 
     def test_uniform(self):
-        low = Variable(torch.zeros(5, 5), requires_grad=True)
-        high = Variable(torch.ones(5, 5) * 3, requires_grad=True)
-        low_1d = Variable(torch.zeros(1), requires_grad=True)
-        high_1d = Variable(torch.ones(1) * 3, requires_grad=True)
+        low = variable(torch.zeros(5, 5), requires_grad=True)
+        high = variable(torch.ones(5, 5) * 3, requires_grad=True)
+        low_1d = variable(torch.zeros(1), requires_grad=True)
+        high_1d = variable(torch.ones(1) * 3, requires_grad=True)
         self.assertEqual(Uniform(low, high).sample().size(), (5, 5))
         self.assertEqual(Uniform(low, high).sample((7,)).size(), (7, 5, 5))
         self.assertEqual(Uniform(low_1d, high_1d).sample().size(), (1,))
@@ -826,8 +1005,8 @@ def test_uniform(self):
 
         # Check log_prob computation when value outside range
         uniform = Uniform(low_1d, high_1d)
-        above_high = Variable(torch.Tensor([4.0]))
-        below_low = Variable(torch.Tensor([-1.0]))
+        above_high = variable([4.0])
+        below_low = variable([-1.0])
         self.assertEqual(uniform.log_prob(above_high).item(), -float('inf'), allow_inf=True)
         self.assertEqual(uniform.log_prob(below_low).item(), -float('inf'), allow_inf=True)
 
@@ -847,10 +1026,10 @@ def test_uniform(self):
         high.grad.zero_()
 
     def test_cauchy(self):
-        loc = Variable(torch.zeros(5, 5), requires_grad=True)
-        scale = Variable(torch.ones(5, 5), requires_grad=True)
-        loc_1d = Variable(torch.zeros(1), requires_grad=True)
-        scale_1d = Variable(torch.ones(1), requires_grad=True)
+        loc = variable(torch.zeros(5, 5), requires_grad=True)
+        scale = variable(torch.ones(5, 5), requires_grad=True)
+        loc_1d = variable(torch.zeros(1), requires_grad=True)
+        scale_1d = variable(torch.ones(1), requires_grad=True)
         self.assertTrue(is_all_nan(Cauchy(loc_1d, scale_1d).mean))
         self.assertEqual(Cauchy(loc_1d, scale_1d).variance, float('inf'), allow_inf=True)
         self.assertEqual(Cauchy(loc, scale).sample().size(), (5, 5))
@@ -875,12 +1054,12 @@ def test_cauchy(self):
         scale.grad.zero_()
 
     def test_lognormal(self):
-        mean = Variable(torch.randn(5, 5), requires_grad=True)
-        std = Variable(torch.randn(5, 5).abs(), requires_grad=True)
-        mean_1d = Variable(torch.randn(1), requires_grad=True)
-        std_1d = Variable(torch.randn(1), requires_grad=True)
-        mean_delta = torch.Tensor([1.0, 0.0])
-        std_delta = torch.Tensor([1e-5, 1e-5])
+        mean = variable(torch.randn(5, 5), requires_grad=True)
+        std = variable(torch.randn(5, 5).abs(), requires_grad=True)
+        mean_1d = variable(torch.randn(1), requires_grad=True)
+        std_1d = variable(torch.randn(1), requires_grad=True)
+        mean_delta = variable([1.0, 0.0])
+        std_delta = variable([1e-5, 1e-5])
         self.assertEqual(LogNormal(mean, std).sample().size(), (5, 5))
         self.assertEqual(LogNormal(mean, std).sample((7,)).size(), (7, 5, 5))
         self.assertEqual(LogNormal(mean_1d, std_1d).sample((1,)).size(), (1, 1))
@@ -900,8 +1079,8 @@ def test_lognormal(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_lognormal_logprob(self):
-        mean = Variable(torch.randn(5, 1), requires_grad=True)
-        std = Variable(torch.randn(5, 1).abs(), requires_grad=True)
+        mean = variable(torch.randn(5, 1), requires_grad=True)
+        std = variable(torch.randn(5, 1).abs(), requires_grad=True)
 
         def ref_log_prob(idx, x, log_prob):
             m = mean.data.view(-1)[idx]
@@ -920,12 +1099,12 @@ def test_lognormal_sample(self):
                                         'LogNormal(loc={}, scale={})'.format(mean, std))
 
     def test_normal(self):
-        loc = Variable(torch.randn(5, 5), requires_grad=True)
-        scale = Variable(torch.randn(5, 5).abs(), requires_grad=True)
-        loc_1d = Variable(torch.randn(1), requires_grad=True)
-        scale_1d = Variable(torch.randn(1), requires_grad=True)
-        loc_delta = torch.Tensor([1.0, 0.0])
-        scale_delta = torch.Tensor([1e-5, 1e-5])
+        loc = variable(torch.randn(5, 5), requires_grad=True)
+        scale = variable(torch.randn(5, 5).abs(), requires_grad=True)
+        loc_1d = variable(torch.randn(1), requires_grad=True)
+        scale_1d = variable(torch.randn(1), requires_grad=True)
+        loc_delta = variable([1.0, 0.0])
+        scale_delta = variable([1e-5, 1e-5])
         self.assertEqual(Normal(loc, scale).sample().size(), (5, 5))
         self.assertEqual(Normal(loc, scale).sample((7,)).size(), (7, 5, 5))
         self.assertEqual(Normal(loc_1d, scale_1d).sample((1,)).size(), (1, 1))
@@ -972,8 +1151,8 @@ def test_normal_sample(self):
                                         'Normal(mean={}, std={})'.format(loc, scale))
 
     def test_exponential(self):
-        rate = Variable(torch.randn(5, 5).abs(), requires_grad=True)
-        rate_1d = Variable(torch.randn(1).abs(), requires_grad=True)
+        rate = variable(torch.randn(5, 5).abs(), requires_grad=True)
+        rate_1d = variable(torch.randn(1).abs(), requires_grad=True)
         self.assertEqual(Exponential(rate).sample().size(), (5, 5))
         self.assertEqual(Exponential(rate).sample((7,)).size(), (7, 5, 5))
         self.assertEqual(Exponential(rate_1d).sample((1,)).size(), (1, 1))
@@ -1007,10 +1186,10 @@ def test_exponential_sample(self):
                                         'Exponential(rate={})'.format(rate))
 
     def test_laplace(self):
-        loc = Variable(torch.randn(5, 5), requires_grad=True)
-        scale = Variable(torch.randn(5, 5).abs(), requires_grad=True)
-        loc_1d = Variable(torch.randn(1), requires_grad=True)
-        scale_1d = Variable(torch.randn(1), requires_grad=True)
+        loc = variable(torch.randn(5, 5), requires_grad=True)
+        scale = variable(torch.randn(5, 5).abs(), requires_grad=True)
+        loc_1d = variable(torch.randn(1), requires_grad=True)
+        scale_1d = variable(torch.randn(1), requires_grad=True)
         loc_delta = torch.Tensor([1.0, 0.0])
         scale_delta = torch.Tensor([1e-5, 1e-5])
         self.assertEqual(Laplace(loc, scale).sample().size(), (5, 5))
@@ -1059,10 +1238,10 @@ def test_laplace_sample(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_gamma_shape(self):
-        alpha = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        beta = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        alpha_1d = Variable(torch.exp(torch.randn(1)), requires_grad=True)
-        beta_1d = Variable(torch.exp(torch.randn(1)), requires_grad=True)
+        alpha = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        beta = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        alpha_1d = variable(torch.exp(torch.randn(1)), requires_grad=True)
+        beta_1d = variable(torch.exp(torch.randn(1)), requires_grad=True)
         self.assertEqual(Gamma(alpha, beta).sample().size(), (2, 3))
         self.assertEqual(Gamma(alpha, beta).sample((5,)).size(), (5, 2, 3))
         self.assertEqual(Gamma(alpha_1d, beta_1d).sample((1,)).size(), (1, 1))
@@ -1088,10 +1267,10 @@ def test_gamma_sample(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_pareto(self):
-        scale = Variable(torch.randn(2, 3).abs(), requires_grad=True)
-        alpha = Variable(torch.randn(2, 3).abs(), requires_grad=True)
-        scale_1d = Variable(torch.randn(1).abs(), requires_grad=True)
-        alpha_1d = Variable(torch.randn(1).abs(), requires_grad=True)
+        scale = variable(torch.randn(2, 3).abs(), requires_grad=True)
+        alpha = variable(torch.randn(2, 3).abs(), requires_grad=True)
+        scale_1d = variable(torch.randn(1).abs(), requires_grad=True)
+        alpha_1d = variable(torch.randn(1).abs(), requires_grad=True)
         self.assertEqual(Pareto(scale_1d, 0.5).mean, float('inf'), allow_inf=True)
         self.assertEqual(Pareto(scale_1d, 0.5).variance, float('inf'), allow_inf=True)
         self.assertEqual(Pareto(scale, alpha).sample().size(), (2, 3))
@@ -1119,10 +1298,10 @@ def test_pareto_sample(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_gumbel(self):
-        loc = Variable(torch.randn(2, 3), requires_grad=True)
-        scale = Variable(torch.randn(2, 3).abs(), requires_grad=True)
-        loc_1d = Variable(torch.randn(1), requires_grad=True)
-        scale_1d = Variable(torch.randn(1).abs(), requires_grad=True)
+        loc = variable(torch.randn(2, 3), requires_grad=True)
+        scale = variable(torch.randn(2, 3).abs(), requires_grad=True)
+        loc_1d = variable(torch.randn(1), requires_grad=True)
+        scale_1d = variable(torch.randn(1).abs(), requires_grad=True)
         self.assertEqual(Gumbel(loc, scale).sample().size(), (2, 3))
         self.assertEqual(Gumbel(loc, scale).sample((5,)).size(), (5, 2, 3))
         self.assertEqual(Gumbel(loc_1d, scale_1d).sample().size(), (1,))
@@ -1148,8 +1327,8 @@ def test_gumbel_sample(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_fishersnedecor(self):
-        df1 = Variable(torch.randn(2, 3).abs(), requires_grad=True)
-        df2 = Variable(torch.randn(2, 3).abs(), requires_grad=True)
+        df1 = variable(torch.randn(2, 3).abs(), requires_grad=True)
+        df2 = variable(torch.randn(2, 3).abs(), requires_grad=True)
         df1_1d = torch.randn(1).abs()
         df2_1d = torch.randn(1).abs()
         self.assertTrue(is_all_nan(FisherSnedecor(1, 2).mean))
@@ -1179,8 +1358,8 @@ def test_fishersnedecor_sample(self):
 
     @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
     def test_chi2_shape(self):
-        df = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        df_1d = Variable(torch.exp(torch.randn(1)), requires_grad=True)
+        df = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        df_1d = variable(torch.exp(torch.randn(1)), requires_grad=True)
         self.assertEqual(Chi2(df).sample().size(), (2, 3))
         self.assertEqual(Chi2(df).sample((5,)).size(), (5, 2, 3))
         self.assertEqual(Chi2(df_1d).sample((1,)).size(), (1, 1))
@@ -1206,8 +1385,8 @@ def test_chi2_sample(self):
 
     @unittest.skipIf(not TEST_NUMPY, "Numpy not found")
     def test_studentT(self):
-        df = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        df_1d = Variable(torch.exp(torch.randn(1)), requires_grad=True)
+        df = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        df_1d = variable(torch.exp(torch.randn(1)), requires_grad=True)
         self.assertTrue(is_all_nan(StudentT(1).mean))
         self.assertTrue(is_all_nan(StudentT(1).variance))
         self.assertEqual(StudentT(2).variance, float('inf'), allow_inf=True)
@@ -1247,8 +1426,8 @@ def test_studentT_log_prob(self):
                 self.assertAlmostEqual(float(actual_log_prob[i]), float(expected_log_prob), places=3)
 
     def test_dirichlet_shape(self):
-        alpha = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        alpha_1d = Variable(torch.exp(torch.randn(4)), requires_grad=True)
+        alpha = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        alpha_1d = variable(torch.exp(torch.randn(4)), requires_grad=True)
         self.assertEqual(Dirichlet(alpha).sample().size(), (2, 3))
         self.assertEqual(Dirichlet(alpha).sample((5,)).size(), (5, 2, 3))
         self.assertEqual(Dirichlet(alpha_1d).sample().size(), (4,))
@@ -1275,10 +1454,10 @@ def test_dirichlet_sample(self):
                                     multivariate=True)
 
     def test_beta_shape(self):
-        con1 = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        con0 = Variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
-        con1_1d = Variable(torch.exp(torch.randn(4)), requires_grad=True)
-        con0_1d = Variable(torch.exp(torch.randn(4)), requires_grad=True)
+        con1 = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        con0 = variable(torch.exp(torch.randn(2, 3)), requires_grad=True)
+        con1_1d = variable(torch.exp(torch.randn(4)), requires_grad=True)
+        con0_1d = variable(torch.exp(torch.randn(4)), requires_grad=True)
         self.assertEqual(Beta(con1, con0).sample().size(), (2, 3))
         self.assertEqual(Beta(con1, con0).sample((5,)).size(), (5, 2, 3))
         self.assertEqual(Beta(con1_1d, con0_1d).sample().size(), (4,))
@@ -1333,7 +1512,7 @@ def test_cdf_log_prob(self):
         for Dist, params in EXAMPLES:
             for i, param in enumerate(params):
                 dist = Dist(**param)
-                samples = Variable(dist.sample().data, requires_grad=True)
+                samples = variable(dist.sample().data, requires_grad=True)
                 try:
                     cdfs = dist.cdf(samples)
                     pdfs = dist.log_prob(samples).exp()
@@ -1502,8 +1681,8 @@ class TestRsample(TestCase):
     def test_gamma(self):
         num_samples = 100
         for alpha in [1e-2, 1e-1, 1e0, 1e1, 1e2, 1e3, 1e4]:
-            alphas = Variable(torch.FloatTensor([alpha] * num_samples), requires_grad=True)
-            betas = Variable(torch.ones(num_samples).type_as(alphas))
+            alphas = variable(torch.FloatTensor([alpha] * num_samples), requires_grad=True)
+            betas = variable(torch.ones(num_samples).type_as(alphas))
             x = Gamma(alphas, betas).rsample()
             x.sum().backward()
             x, ind = x.data.sort()
@@ -1531,7 +1710,7 @@ def test_gamma(self):
     def test_chi2(self):
         num_samples = 100
         for df in [1e-2, 1e-1, 1e0, 1e1, 1e2, 1e3, 1e4]:
-            dfs = Variable(torch.FloatTensor([df] * num_samples), requires_grad=True)
+            dfs = variable(torch.FloatTensor([df] * num_samples), requires_grad=True)
             x = Chi2(dfs).rsample()
             x.sum().backward()
             x, ind = x.data.sort()
@@ -1559,7 +1738,7 @@ def test_dirichlet_on_diagonal(self):
         num_samples = 20
         grid = [1e-1, 1e0, 1e1]
         for a0, a1, a2 in product(grid, grid, grid):
-            alphas = Variable(torch.FloatTensor([[a0, a1, a2]] * num_samples), requires_grad=True)
+            alphas = variable(torch.FloatTensor([[a0, a1, a2]] * num_samples), requires_grad=True)
             x = Dirichlet(alphas).rsample()[:, 0]
             x.sum().backward()
             x, ind = x.data.sort()
@@ -1590,8 +1769,8 @@ def test_beta_wrt_alpha(self):
         num_samples = 20
         grid = [1e-2, 1e-1, 1e0, 1e1, 1e2]
         for con1, con0 in product(grid, grid):
-            con1s = Variable(torch.FloatTensor([con1] * num_samples), requires_grad=True)
-            con0s = Variable(torch.FloatTensor([con0] * num_samples).type_as(con1s))
+            con1s = variable(torch.FloatTensor([con1] * num_samples), requires_grad=True)
+            con0s = variable(torch.FloatTensor([con0] * num_samples).type_as(con1s))
             x = Beta(con1s, con0s).rsample()
             x.sum().backward()
             x, ind = x.data.sort()
@@ -1620,8 +1799,8 @@ def test_beta_wrt_beta(self):
         num_samples = 20
         grid = [1e-2, 1e-1, 1e0, 1e1, 1e2]
         for con1, con0 in product(grid, grid):
-            con0s = Variable(torch.FloatTensor([con0] * num_samples), requires_grad=True)
-            con1s = Variable(torch.FloatTensor([con1] * num_samples).type_as(con0s))
+            con0s = variable(torch.FloatTensor([con0] * num_samples), requires_grad=True)
+            con1s = variable(torch.FloatTensor([con1] * num_samples).type_as(con0s))
             x = Beta(con1s, con0s).rsample()
             x.sum().backward()
             x, ind = x.data.sort()
@@ -1650,7 +1829,7 @@ def test_dirichlet_multivariate(self):
         num_samples = 100000
         for shift in [-0.1, -0.05, -0.01, 0.0, 0.01, 0.05, 0.10]:
             alpha = alpha_crit + shift
-            alpha = Variable(torch.FloatTensor([alpha]), requires_grad=True)
+            alpha = variable(torch.FloatTensor([alpha]), requires_grad=True)
             alpha_vec = torch.cat([alpha, alpha, alpha.new([1])])
             z = Dirichlet(alpha_vec.expand(num_samples, 3)).rsample()
             mean_z3 = 1.0 / (2.0 * alpha + 1.0)
@@ -1679,7 +1858,7 @@ def compute_v(x, alpha):
             ], dim=-1)
 
         for a1, a2, a3 in product(alpha_grid, alpha_grid, alpha_grid):
-            alpha = Variable(torch.Tensor([a1, a2, a3]).expand(num_samples, 3), requires_grad=True)
+            alpha = variable([a1, a2, a3], requires_grad=True).expand(num_samples, 3)
             x = Dirichlet(alpha).rsample()
             dlogp_da = grad([Dirichlet(alpha).log_prob(x.detach()).sum()],
                             [alpha], retain_graph=True)[0].data[:, 0]
@@ -1711,13 +1890,18 @@ class TestDistributionShapes(TestCase):
     def setUp(self):
         super(TestCase, self).setUp()
         self.scalar_sample = 1
-        self.tensor_sample_1 = Variable(torch.ones(3, 2))
-        self.tensor_sample_2 = Variable(torch.ones(3, 2, 3))
+        self.tensor_sample_1 = variable(torch.ones(3, 2))
+        self.tensor_sample_2 = variable(torch.ones(3, 2, 3))
+        Distribution.set_default_validate_args(True)
+
+    def tearDown(self):
+        super(TestCase, self).tearDown()
+        Distribution.set_default_validate_args(False)
 
     def test_entropy_shape(self):
         for Dist, params in EXAMPLES:
             for i, param in enumerate(params):
-                dist = Dist(**param)
+                dist = Dist(validate_args=False, **param)
                 try:
                     actual_shape = dist.entropy().size()
                     expected_shape = dist.batch_shape if dist.batch_shape else torch.Size()
@@ -1745,7 +1929,7 @@ def test_bernoulli_shape_tensor_params(self):
         self.assertEqual(bernoulli.sample((3, 2)).size(), torch.Size((3, 2, 3, 2)))
         self.assertEqual(bernoulli.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, bernoulli.log_prob, self.tensor_sample_2)
-        self.assertEqual(bernoulli.log_prob(Variable(torch.ones(3, 1, 1))).size(), torch.Size((3, 3, 2)))
+        self.assertEqual(bernoulli.log_prob(variable(torch.ones(3, 1, 1))).size(), torch.Size((3, 3, 2)))
 
     def test_geometric_shape_scalar_params(self):
         geometric = Geometric(0.3)
@@ -1765,7 +1949,7 @@ def test_geometric_shape_tensor_params(self):
         self.assertEqual(geometric.sample((3, 2)).size(), torch.Size((3, 2, 3, 2)))
         self.assertEqual(geometric.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, geometric.log_prob, self.tensor_sample_2)
-        self.assertEqual(geometric.log_prob(Variable(torch.ones(3, 1, 1))).size(), torch.Size((3, 3, 2)))
+        self.assertEqual(geometric.log_prob(variable(torch.ones(3, 1, 1))).size(), torch.Size((3, 3, 2)))
 
     def test_beta_shape_scalar_params(self):
         dist = Beta(0.1, 0.1)
@@ -1786,7 +1970,7 @@ def test_beta_shape_tensor_params(self):
         self.assertEqual(dist.sample((3, 2)).size(), torch.Size((3, 2, 3, 2)))
         self.assertEqual(dist.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, dist.log_prob, self.tensor_sample_2)
-        self.assertEqual(dist.log_prob(Variable(torch.ones(3, 1, 1))).size(), torch.Size((3, 3, 2)))
+        self.assertEqual(dist.log_prob(variable(torch.ones(3, 1, 1))).size(), torch.Size((3, 3, 2)))
 
     def test_binomial_shape(self):
         dist = Binomial(10, torch.tensor([0.6, 0.3]))
@@ -1805,7 +1989,7 @@ def test_multinomial_shape(self):
         self.assertEqual(dist.sample((3, 2)).size(), torch.Size((3, 2, 3, 2)))
         self.assertEqual(dist.log_prob(self.tensor_sample_1).size(), torch.Size((3,)))
         self.assertRaises(ValueError, dist.log_prob, self.tensor_sample_2)
-        self.assertEqual(dist.log_prob(Variable(torch.ones(3, 1, 2))).size(), torch.Size((3, 3)))
+        self.assertEqual(dist.log_prob(variable(torch.ones(3, 1, 2))).size(), torch.Size((3, 3)))
 
     def test_categorical_shape(self):
         # unbatched
@@ -1816,7 +2000,7 @@ def test_categorical_shape(self):
         self.assertEqual(dist.sample((3, 2)).size(), torch.Size((3, 2,)))
         self.assertEqual(dist.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertEqual(dist.log_prob(self.tensor_sample_2).size(), torch.Size((3, 2, 3)))
-        self.assertEqual(dist.log_prob(Variable(torch.ones(3, 1))).size(), torch.Size((3, 1)))
+        self.assertEqual(dist.log_prob(variable(torch.ones(3, 1))).size(), torch.Size((3, 1)))
         # batched
         dist = Categorical(torch.tensor([[0.6, 0.3], [0.6, 0.3], [0.6, 0.3]]))
         self.assertEqual(dist._batch_shape, torch.Size((3,)))
@@ -1825,7 +2009,7 @@ def test_categorical_shape(self):
         self.assertEqual(dist.sample((3, 2)).size(), torch.Size((3, 2, 3,)))
         self.assertRaises(ValueError, dist.log_prob, self.tensor_sample_1)
         self.assertEqual(dist.log_prob(self.tensor_sample_2).size(), torch.Size((3, 2, 3)))
-        self.assertEqual(dist.log_prob(Variable(torch.ones(3, 1))).size(), torch.Size((3, 3)))
+        self.assertEqual(dist.log_prob(variable(torch.ones(3, 1))).size(), torch.Size((3, 3)))
 
     def test_one_hot_categorical_shape(self):
         # unbatched
@@ -1835,19 +2019,23 @@ def test_one_hot_categorical_shape(self):
         self.assertEqual(dist.sample().size(), torch.Size((3,)))
         self.assertEqual(dist.sample((3, 2)).size(), torch.Size((3, 2, 3)))
         self.assertRaises(ValueError, dist.log_prob, self.tensor_sample_1)
-        self.assertEqual(dist.log_prob(self.tensor_sample_2).size(), torch.Size((3, 2,)))
+        simplex_sample = self.tensor_sample_2 / self.tensor_sample_2.sum(-1, keepdim=True)
+        self.assertEqual(dist.log_prob(simplex_sample).size(), torch.Size((3, 2,)))
         self.assertEqual(dist.log_prob(dist.enumerate_support()).size(), torch.Size((3,)))
-        self.assertEqual(dist.log_prob(Variable(torch.ones(3, 3))).size(), torch.Size((3,)))
+        simplex_sample = variable(torch.ones(3, 3)) / 3
+        self.assertEqual(dist.log_prob(simplex_sample).size(), torch.Size((3,)))
         # batched
         dist = OneHotCategorical(torch.tensor([[0.6, 0.3], [0.6, 0.3], [0.6, 0.3]]))
         self.assertEqual(dist._batch_shape, torch.Size((3,)))
         self.assertEqual(dist._event_shape, torch.Size((2,)))
         self.assertEqual(dist.sample().size(), torch.Size((3, 2)))
         self.assertEqual(dist.sample((3, 2)).size(), torch.Size((3, 2, 3, 2)))
-        self.assertEqual(dist.log_prob(self.tensor_sample_1).size(), torch.Size((3,)))
+        simplex_sample = self.tensor_sample_1 / self.tensor_sample_1.sum(-1, keepdim=True)
+        self.assertEqual(dist.log_prob(simplex_sample).size(), torch.Size((3,)))
         self.assertRaises(ValueError, dist.log_prob, self.tensor_sample_2)
         self.assertEqual(dist.log_prob(dist.enumerate_support()).size(), torch.Size((2, 3)))
-        self.assertEqual(dist.log_prob(Variable(torch.ones((3, 1, 2)))).size(), torch.Size((3, 3)))
+        simplex_sample = variable(torch.ones(3, 1, 2)) / 2
+        self.assertEqual(dist.log_prob(simplex_sample).size(), torch.Size((3, 3)))
 
     def test_cauchy_shape_scalar_params(self):
         cauchy = Cauchy(0, 1)
@@ -1867,7 +2055,7 @@ def test_cauchy_shape_tensor_params(self):
         self.assertEqual(cauchy.sample(torch.Size((3, 2))).size(), torch.Size((3, 2, 2)))
         self.assertEqual(cauchy.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, cauchy.log_prob, self.tensor_sample_2)
-        self.assertEqual(cauchy.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(cauchy.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
     def test_dirichlet_shape(self):
         dist = Dirichlet(torch.tensor([[0.6, 0.3], [1.6, 1.3], [2.6, 2.3]]))
@@ -1875,9 +2063,12 @@ def test_dirichlet_shape(self):
         self.assertEqual(dist._event_shape, torch.Size((2,)))
         self.assertEqual(dist.sample().size(), torch.Size((3, 2)))
         self.assertEqual(dist.sample((5, 4)).size(), torch.Size((5, 4, 3, 2)))
-        self.assertEqual(dist.log_prob(self.tensor_sample_1).size(), torch.Size((3,)))
+        simplex_sample = self.tensor_sample_1 / self.tensor_sample_1.sum(-1, keepdim=True)
+        self.assertEqual(dist.log_prob(simplex_sample).size(), torch.Size((3,)))
         self.assertRaises(ValueError, dist.log_prob, self.tensor_sample_2)
-        self.assertEqual(dist.log_prob(Variable(torch.ones((3, 1, 2)))).size(), torch.Size((3, 3)))
+        simplex_sample = torch.ones((3, 1, 2))
+        simplex_sample = simplex_sample / simplex_sample.sum(-1).unsqueeze(-1)
+        self.assertEqual(dist.log_prob(simplex_sample).size(), torch.Size((3, 3)))
 
     def test_gamma_shape_scalar_params(self):
         gamma = Gamma(1, 1)
@@ -1897,7 +2088,7 @@ def test_gamma_shape_tensor_params(self):
         self.assertEqual(gamma.sample((3, 2)).size(), torch.Size((3, 2, 2)))
         self.assertEqual(gamma.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, gamma.log_prob, self.tensor_sample_2)
-        self.assertEqual(gamma.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(gamma.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
     def test_chi2_shape_scalar_params(self):
         chi2 = Chi2(1)
@@ -1917,7 +2108,7 @@ def test_chi2_shape_tensor_params(self):
         self.assertEqual(chi2.sample((3, 2)).size(), torch.Size((3, 2, 2)))
         self.assertEqual(chi2.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, chi2.log_prob, self.tensor_sample_2)
-        self.assertEqual(chi2.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(chi2.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
     def test_studentT_shape_scalar_params(self):
         st = StudentT(1)
@@ -1937,7 +2128,7 @@ def test_studentT_shape_tensor_params(self):
         self.assertEqual(st.sample((3, 2)).size(), torch.Size((3, 2, 2)))
         self.assertEqual(st.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, st.log_prob, self.tensor_sample_2)
-        self.assertEqual(st.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(st.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
     def test_pareto_shape_scalar_params(self):
         pareto = Pareto(1, 1)
@@ -1945,9 +2136,8 @@ def test_pareto_shape_scalar_params(self):
         self.assertEqual(pareto._event_shape, torch.Size())
         self.assertEqual(pareto.sample().size(), torch.Size())
         self.assertEqual(pareto.sample((3, 2)).size(), torch.Size((3, 2)))
-        self.assertRaises(ValueError, pareto.log_prob, self.scalar_sample)
-        self.assertEqual(pareto.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
-        self.assertEqual(pareto.log_prob(self.tensor_sample_2).size(), torch.Size((3, 2, 3)))
+        self.assertEqual(pareto.log_prob(self.tensor_sample_1 + 1).size(), torch.Size((3, 2)))
+        self.assertEqual(pareto.log_prob(self.tensor_sample_2 + 1).size(), torch.Size((3, 2, 3)))
 
     def test_gumbel_shape_scalar_params(self):
         gumbel = Gumbel(1, 1)
@@ -1955,7 +2145,6 @@ def test_gumbel_shape_scalar_params(self):
         self.assertEqual(gumbel._event_shape, torch.Size())
         self.assertEqual(gumbel.sample().size(), torch.Size())
         self.assertEqual(gumbel.sample((3, 2)).size(), torch.Size((3, 2)))
-        self.assertRaises(ValueError, gumbel.log_prob, self.scalar_sample)
         self.assertEqual(gumbel.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertEqual(gumbel.log_prob(self.tensor_sample_2).size(), torch.Size((3, 2, 3)))
 
@@ -1977,7 +2166,7 @@ def test_normal_shape_tensor_params(self):
         self.assertEqual(normal.sample((3, 2)).size(), torch.Size((3, 2, 2)))
         self.assertEqual(normal.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, normal.log_prob, self.tensor_sample_2)
-        self.assertEqual(normal.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(normal.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
     def test_uniform_shape_scalar_params(self):
         uniform = Uniform(0, 1)
@@ -1997,7 +2186,7 @@ def test_uniform_shape_tensor_params(self):
         self.assertEqual(uniform.sample(torch.Size((3, 2))).size(), torch.Size((3, 2, 2)))
         self.assertEqual(uniform.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, uniform.log_prob, self.tensor_sample_2)
-        self.assertEqual(uniform.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(uniform.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
     def test_exponential_shape_scalar_param(self):
         expon = Exponential(1.)
@@ -2017,7 +2206,7 @@ def test_exponential_shape_tensor_param(self):
         self.assertEqual(expon.sample((3, 2)).size(), torch.Size((3, 2, 2)))
         self.assertEqual(expon.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, expon.log_prob, self.tensor_sample_2)
-        self.assertEqual(expon.log_prob(Variable(torch.ones(2, 2))).size(), torch.Size((2, 2)))
+        self.assertEqual(expon.log_prob(variable(torch.ones(2, 2))).size(), torch.Size((2, 2)))
 
     def test_laplace_shape_scalar_params(self):
         laplace = Laplace(0, 1)
@@ -2037,7 +2226,7 @@ def test_laplace_shape_tensor_params(self):
         self.assertEqual(laplace.sample((3, 2)).size(), torch.Size((3, 2, 2)))
         self.assertEqual(laplace.log_prob(self.tensor_sample_1).size(), torch.Size((3, 2)))
         self.assertRaises(ValueError, laplace.log_prob, self.tensor_sample_2)
-        self.assertEqual(laplace.log_prob(Variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
+        self.assertEqual(laplace.log_prob(variable(torch.ones(2, 1))).size(), torch.Size((2, 2)))
 
 
 class TestKL(TestCase):
@@ -2312,8 +2501,12 @@ def test_params_contains(self):
                         constraint = dist.params[name]
                     except KeyError:
                         continue  # ignore optional parameters
+                    except TypeError:
+                        continue
+
                     if is_dependent(constraint):
                         continue
+
                     message = '{} example {}/{} parameter {} = {}'.format(
                         Dist.__name__, i + 1, len(params), name, value)
                     self.assertTrue(constraint.check(value).all(), msg=message)
@@ -2340,12 +2533,12 @@ def _test_pdf_score(self,
                         expected_gradient=None,
                         prec=1e-5):
         if probs is not None:
-            p = Variable(probs, requires_grad=True)
+            p = variable(probs, requires_grad=True)
             dist = dist_class(p)
         else:
-            p = Variable(logits, requires_grad=True)
+            p = variable(logits, requires_grad=True)
             dist = dist_class(logits=p)
-        log_pdf = dist.log_prob(Variable(x))
+        log_pdf = dist.log_prob(variable(x))
         log_pdf.sum().backward()
         self.assertEqual(log_pdf.data,
                          expected_value,
@@ -2417,35 +2610,35 @@ def test_bernoulli_with_logits_overflow(self):
 
     def test_categorical_log_prob(self):
         for tensor_type in ([torch.FloatTensor, torch.DoubleTensor]):
-            p = Variable(tensor_type([0, 1]), requires_grad=True)
+            p = variable(tensor_type([0, 1]), requires_grad=True)
             categorical = OneHotCategorical(p)
-            log_pdf = categorical.log_prob(Variable(tensor_type([0, 1])))
+            log_pdf = categorical.log_prob(variable(tensor_type([0, 1])))
             self.assertEqual(log_pdf.item(), 0)
 
     def test_categorical_log_prob_with_logits(self):
         for tensor_type in ([torch.FloatTensor, torch.DoubleTensor]):
-            p = Variable(tensor_type([-float('inf'), 0]), requires_grad=True)
+            p = variable(tensor_type([-float('inf'), 0]), requires_grad=True)
             categorical = OneHotCategorical(logits=p)
-            log_pdf_prob_1 = categorical.log_prob(Variable(tensor_type([0, 1])))
+            log_pdf_prob_1 = categorical.log_prob(variable(tensor_type([0, 1])))
             self.assertEqual(log_pdf_prob_1.item(), 0)
-            log_pdf_prob_0 = categorical.log_prob(Variable(tensor_type([1, 0])))
+            log_pdf_prob_0 = categorical.log_prob(variable(tensor_type([1, 0])))
             self.assertEqual(log_pdf_prob_0.item(), -float('inf'), allow_inf=True)
 
     def test_multinomial_log_prob(self):
         for tensor_type in [torch.FloatTensor, torch.DoubleTensor]:
-            p = Variable(tensor_type([0, 1]), requires_grad=True)
-            s = Variable(tensor_type([0, 10]))
+            p = variable(tensor_type([0, 1]), requires_grad=True)
+            s = variable(tensor_type([0, 10]))
             multinomial = Multinomial(10, p)
             log_pdf = multinomial.log_prob(s)
             self.assertEqual(log_pdf.item(), 0)
 
     def test_multinomial_log_prob_with_logits(self):
         for tensor_type in [torch.FloatTensor, torch.DoubleTensor]:
-            p = Variable(tensor_type([-float('inf'), 0]), requires_grad=True)
+            p = variable(tensor_type([-float('inf'), 0]), requires_grad=True)
             multinomial = Multinomial(10, logits=p)
-            log_pdf_prob_1 = multinomial.log_prob(Variable(tensor_type([0, 10])))
+            log_pdf_prob_1 = multinomial.log_prob(variable(tensor_type([0, 10])))
             self.assertEqual(log_pdf_prob_1.item(), 0)
-            log_pdf_prob_0 = multinomial.log_prob(Variable(tensor_type([10, 0])))
+            log_pdf_prob_0 = multinomial.log_prob(variable(tensor_type([10, 0])))
             self.assertEqual(log_pdf_prob_0.item(), -float('inf'), allow_inf=True)
 
 
@@ -2462,7 +2655,7 @@ def test_lazy_logits_initialization(self):
                 param['logits'] = probs_to_logits(probs)
                 dist = Dist(**param)
                 shape = (1,) if not dist.event_shape else dist.event_shape
-                dist.log_prob(Variable(torch.ones(shape)))
+                dist.log_prob(variable(torch.ones(shape)))
                 message = 'Failed for {} example 0/{}'.format(Dist.__name__, len(params))
                 self.assertFalse('probs' in vars(dist), msg=message)
                 try:
@@ -2493,9 +2686,9 @@ def test_lazy_probs_initialization(self):
 @unittest.skipIf(not TEST_NUMPY, "NumPy not found")
 class TestAgainstScipy(TestCase):
     def setUp(self):
-        positive_var = Variable(torch.Tensor(20,).normal_()).exp()
-        positive_var2 = Variable(torch.Tensor(20,).normal_()).exp()
-        random_var = Variable(torch.Tensor(20,).normal_())
+        positive_var = variable(torch.Tensor(20,).normal_()).exp()
+        positive_var2 = variable(torch.Tensor(20,).normal_()).exp()
+        random_var = variable(torch.Tensor(20,).normal_())
         random_tensor = torch.Tensor(20,).normal_()
         simplex_tensor = softmax(random_tensor)
         self.distribution_pairs = [
@@ -2606,7 +2799,7 @@ def test_cdf(self):
 
     def test_icdf(self):
         for pytorch_dist, scipy_dist in self.distribution_pairs:
-            samples = Variable(torch.rand((5,) + pytorch_dist.batch_shape))
+            samples = variable(torch.rand((5,) + pytorch_dist.batch_shape))
             try:
                 icdf = pytorch_dist.icdf(samples)
             except NotImplementedError:
@@ -2623,23 +2816,23 @@ def setUp(self):
                 AbsTransform(cache_size=cache_size),
                 ExpTransform(cache_size=cache_size),
                 SigmoidTransform(cache_size=cache_size),
-                AffineTransform(Variable(torch.Tensor(5).normal_()),
-                                Variable(torch.Tensor(5).normal_()),
+                AffineTransform(variable(torch.Tensor(5).normal_()),
+                                variable(torch.Tensor(5).normal_()),
                                 cache_size=cache_size),
-                AffineTransform(Variable(torch.Tensor(4, 5).normal_()),
-                                Variable(torch.Tensor(4, 5).normal_()),
+                AffineTransform(variable(torch.Tensor(4, 5).normal_()),
+                                variable(torch.Tensor(4, 5).normal_()),
                                 cache_size=cache_size),
                 BoltzmannTransform(cache_size=cache_size),
                 StickBreakingTransform(cache_size=cache_size),
                 LowerCholeskyTransform(cache_size=cache_size),
                 ComposeTransform([
-                    AffineTransform(Variable(torch.Tensor(4, 5).normal_()),
-                                    Variable(torch.Tensor(4, 5).normal_()),
+                    AffineTransform(variable(torch.Tensor(4, 5).normal_()),
+                                    variable(torch.Tensor(4, 5).normal_()),
                                     cache_size=cache_size),
                 ]),
                 ComposeTransform([
-                    AffineTransform(Variable(torch.Tensor(4, 5).normal_()),
-                                    Variable(torch.Tensor(4, 5).normal_()),
+                    AffineTransform(variable(torch.Tensor(4, 5).normal_()),
+                                    variable(torch.Tensor(4, 5).normal_()),
                                     cache_size=cache_size),
                     ExpTransform(cache_size=cache_size),
                 ]),
@@ -2690,7 +2883,7 @@ def test_equality(self):
 
     def test_forward_inverse_cache(self):
         for transform in self.transforms:
-            x = Variable(self._generate_data(transform), requires_grad=True)
+            x = variable(self._generate_data(transform), requires_grad=True)
             try:
                 y = transform(x)
             except NotImplementedError:
@@ -2717,7 +2910,7 @@ def test_forward_inverse_cache(self):
 
     def test_forward_inverse_no_cache(self):
         for transform in self.transforms:
-            x = Variable(self._generate_data(transform), requires_grad=True)
+            x = variable(self._generate_data(transform), requires_grad=True)
             try:
                 y = transform(x)
                 x2 = transform.inv(y.clone())  # bypass cache
@@ -2744,7 +2937,7 @@ def test_forward_inverse_no_cache(self):
 
     def test_univariate_forward_jacobian(self):
         for transform in self.transforms:
-            x = Variable(self._generate_data(transform), requires_grad=True)
+            x = variable(self._generate_data(transform), requires_grad=True)
             try:
                 y = transform(x)
                 actual = transform.log_abs_det_jacobian(x, y)
@@ -2759,7 +2952,7 @@ def test_univariate_forward_jacobian(self):
 
     def test_univariate_inverse_jacobian(self):
         for transform in self.transforms:
-            y = Variable(self._generate_data(transform.inv), requires_grad=True)
+            y = variable(self._generate_data(transform.inv), requires_grad=True)
             try:
                 x = transform.inv(y)
                 actual = transform.log_abs_det_jacobian(x, y)
@@ -2788,8 +2981,8 @@ def test_transformed_distribution_shapes(self):
         transform0 = ExpTransform()
         transform1 = BoltzmannTransform()
         transform2 = LowerCholeskyTransform()
-        base_dist0 = Normal(Variable(torch.zeros(4, 4)), Variable(torch.ones(4, 4)))
-        base_dist1 = Dirichlet(Variable(torch.ones(4, 4)))
+        base_dist0 = Normal(variable(torch.zeros(4, 4)), variable(torch.ones(4, 4)))
+        base_dist1 = Dirichlet(variable(torch.ones(4, 4)))
         examples = [
             ((4, 4), (), base_dist0),
             ((4,), (4,), base_dist1),
@@ -2842,7 +3035,7 @@ def test_biject_to(self):
             except NotImplementedError:
                 continue
             self.assertTrue(t.bijective, "biject_to({}) is not bijective".format(constraint))
-            x = Variable(torch.Tensor(5, 5)).normal_()
+            x = variable(torch.Tensor(5, 5)).normal_()
             y = t(x)
             self.assertTrue(constraint.check(y).all(), '\n'.join([
                 "Failed to biject_to({})".format(constraint),
@@ -2855,7 +3048,7 @@ def test_biject_to(self):
     def test_transform_to(self):
         for constraint in self.constraints:
             t = transform_to(constraint)
-            x = Variable(torch.Tensor(5, 5)).normal_()
+            x = variable(torch.Tensor(5, 5)).normal_()
             y = t(x)
             self.assertTrue(constraint.check(y).all(), "Failed to transform_to({})".format(constraint))
             x2 = t.inv(y)
@@ -2863,5 +3056,29 @@ def test_transform_to(self):
             self.assertEqual(y, y2, message="Error in transform_to({}) pseudoinverse".format(constraint))
 
 
+class TestValidation(TestCase):
+    def setUp(self):
+        super(TestCase, self).setUp()
+        Distribution.set_default_validate_args(True)
+
+    def test_valid(self):
+        for Dist, params in EXAMPLES:
+            for i, param in enumerate(params):
+                Dist(validate_args=True, **param)
+
+    def test_invalid(self):
+        for Dist, params in BAD_EXAMPLES:
+            for i, param in enumerate(params):
+                try:
+                    with self.assertRaises(ValueError):
+                        Dist(validate_args=True, **param)
+                except AssertionError:
+                    fail_string = 'ValueError not raised for {} example {}/{}'
+                    raise AssertionError(fail_string.format(Dist.__name__, i + 1, len(params)))
+
+    def tearDown(self):
+        super(TestCase, self).tearDown()
+        Distribution.set_default_validate_args(False)
+
 if __name__ == '__main__':
     run_tests()
diff --git a/torch/__init__.py b/torch/__init__.py
index f9582b1ca609..c9a5f2d33d88 100644
--- a/torch/__init__.py
+++ b/torch/__init__.py
@@ -173,7 +173,6 @@ def set_default_tensor_type(t):
         Tensor = _import_dotted_name(t)
         _C._set_default_tensor_type(Tensor)
 
-
 from .random import set_rng_state, get_rng_state, manual_seed, initial_seed
 from .serialization import save, load
 from ._tensor_str import set_printoptions
diff --git a/torch/distributions/bernoulli.py b/torch/distributions/bernoulli.py
index 172bd5518ed0..cc6e72dab573 100644
--- a/torch/distributions/bernoulli.py
+++ b/torch/distributions/bernoulli.py
@@ -31,7 +31,7 @@ class Bernoulli(ExponentialFamily):
     has_enumerate_support = True
     _mean_carrier_measure = 0
 
-    def __init__(self, probs=None, logits=None):
+    def __init__(self, probs=None, logits=None, validate_args=None):
         if (probs is None) == (logits is None):
             raise ValueError("Either `probs` or `logits` must be specified, but not both.")
         if probs is not None:
@@ -45,7 +45,7 @@ def __init__(self, probs=None, logits=None):
             batch_shape = torch.Size()
         else:
             batch_shape = self._param.size()
-        super(Bernoulli, self).__init__(batch_shape)
+        super(Bernoulli, self).__init__(batch_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._param.new(*args, **kwargs)
@@ -76,7 +76,8 @@ def sample(self, sample_shape=torch.Size()):
             return torch.bernoulli(self.probs.expand(shape))
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         logits, value = broadcast_all(self.logits, value)
         return -binary_cross_entropy_with_logits(logits, value, reduce=False)
 
diff --git a/torch/distributions/beta.py b/torch/distributions/beta.py
index dce00f1c3c55..85e1022142da 100644
--- a/torch/distributions/beta.py
+++ b/torch/distributions/beta.py
@@ -28,14 +28,14 @@ class Beta(ExponentialFamily):
     support = constraints.unit_interval
     has_rsample = True
 
-    def __init__(self, concentration1, concentration0):
+    def __init__(self, concentration1, concentration0, validate_args=None):
         if isinstance(concentration1, Number) and isinstance(concentration0, Number):
             concentration1_concentration0 = torch.tensor([concentration1, concentration0])
         else:
             concentration1, concentration0 = broadcast_all(concentration1, concentration0)
             concentration1_concentration0 = torch.stack([concentration1, concentration0], -1)
         self._dirichlet = Dirichlet(concentration1_concentration0)
-        super(Beta, self).__init__(self._dirichlet._batch_shape)
+        super(Beta, self).__init__(self._dirichlet._batch_shape, validate_args=validate_args)
 
     @property
     def mean(self):
@@ -54,7 +54,8 @@ def rsample(self, sample_shape=()):
         return value
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         heads_tails = torch.stack([value, 1.0 - value], -1)
         return self._dirichlet.log_prob(heads_tails)
 
diff --git a/torch/distributions/binomial.py b/torch/distributions/binomial.py
index 3c7e0208430c..590e5f1dbd45 100644
--- a/torch/distributions/binomial.py
+++ b/torch/distributions/binomial.py
@@ -34,7 +34,7 @@ class Binomial(Distribution):
     params = {'probs': constraints.unit_interval}
     has_enumerate_support = True
 
-    def __init__(self, total_count=1, probs=None, logits=None):
+    def __init__(self, total_count=1, probs=None, logits=None, validate_args=None):
         if not isinstance(total_count, Number):
             raise NotImplementedError('inhomogeneous total_count is not supported')
         self.total_count = total_count
@@ -52,7 +52,7 @@ def __init__(self, total_count=1, probs=None, logits=None):
             batch_shape = torch.Size()
         else:
             batch_shape = self._param.size()
-        super(Binomial, self).__init__(batch_shape)
+        super(Binomial, self).__init__(batch_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._param.new(*args, **kwargs)
@@ -87,7 +87,8 @@ def sample(self, sample_shape=torch.Size()):
             return torch.bernoulli(self.probs.unsqueeze(-1).expand(shape)).sum(dim=-1)
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         log_factorial_n = math.lgamma(self.total_count + 1)
         log_factorial_k = torch.lgamma(value + 1)
         log_factorial_nmk = torch.lgamma(self.total_count - value + 1)
diff --git a/torch/distributions/categorical.py b/torch/distributions/categorical.py
index f88fa9ebd25f..3ed0593e2ff3 100644
--- a/torch/distributions/categorical.py
+++ b/torch/distributions/categorical.py
@@ -40,7 +40,7 @@ class Categorical(Distribution):
     params = {'probs': constraints.simplex}
     has_enumerate_support = True
 
-    def __init__(self, probs=None, logits=None):
+    def __init__(self, probs=None, logits=None, validate_args=None):
         if (probs is None) == (logits is None):
             raise ValueError("Either `probs` or `logits` must be specified, but not both.")
         if probs is not None:
@@ -49,8 +49,8 @@ def __init__(self, probs=None, logits=None):
             self.logits = logits - log_sum_exp(logits)
         self._param = self.probs if probs is not None else self.logits
         self._num_events = self._param.size()[-1]
-        batch_shape = self._param.size()[:-1]
-        super(Categorical, self).__init__(batch_shape)
+        batch_shape = self._param.size()[:-1] if self._param.ndimension() > 1 else torch.Size()
+        super(Categorical, self).__init__(batch_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._param.new(*args, **kwargs)
@@ -91,7 +91,8 @@ def sample(self, sample_shape=torch.Size()):
         return sample_2d.contiguous().view(sample_shape)
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         value_shape = torch._C._infer_size(value.size(), self.batch_shape) if self.batch_shape else value.size()
         param_shape = value_shape + (self._num_events,)
         value = value.expand(value_shape)
diff --git a/torch/distributions/cauchy.py b/torch/distributions/cauchy.py
index e3689c84b857..74f7009203fe 100644
--- a/torch/distributions/cauchy.py
+++ b/torch/distributions/cauchy.py
@@ -28,13 +28,13 @@ class Cauchy(Distribution):
     support = constraints.real
     has_rsample = True
 
-    def __init__(self, loc, scale):
+    def __init__(self, loc, scale, validate_args=None):
         self.loc, self.scale = broadcast_all(loc, scale)
         if isinstance(loc, Number) and isinstance(scale, Number):
             batch_shape = torch.Size()
         else:
             batch_shape = self.loc.size()
-        super(Cauchy, self).__init__(batch_shape)
+        super(Cauchy, self).__init__(batch_shape, validate_args=validate_args)
 
     @property
     def mean(self):
@@ -50,15 +50,18 @@ def rsample(self, sample_shape=torch.Size()):
         return self.loc + eps * self.scale
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return -math.log(math.pi) - self.scale.log() - (1 + ((value - self.loc) / self.scale)**2).log()
 
     def cdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return torch.atan((value - self.loc) / self.scale) / math.pi + 0.5
 
     def icdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return torch.tan(math.pi * (value - 0.5)) * self.scale + self.loc
 
     def entropy(self):
diff --git a/torch/distributions/chi2.py b/torch/distributions/chi2.py
index f107d8b0268f..73fb325fdfa9 100644
--- a/torch/distributions/chi2.py
+++ b/torch/distributions/chi2.py
@@ -19,5 +19,9 @@ class Chi2(Gamma):
     """
     params = {'df': constraints.positive}
 
-    def __init__(self, df):
-        super(Chi2, self).__init__(0.5 * df, 0.5)
+    def __init__(self, df, validate_args=None):
+        super(Chi2, self).__init__(0.5 * df, 0.5, validate_args=validate_args)
+
+    @property
+    def df(self):
+        return self.concentration * 2
diff --git a/torch/distributions/constraints.py b/torch/distributions/constraints.py
index 6100251377df..daa228e55a5c 100644
--- a/torch/distributions/constraints.py
+++ b/torch/distributions/constraints.py
@@ -139,24 +139,24 @@ def check(self, value):
 
 class _GreaterThan(Constraint):
     """
-    Constrain to a real half line `[lower_bound, inf]`.
+    Constrain to a real half line `(lower_bound, inf]`.
     """
     def __init__(self, lower_bound):
         self.lower_bound = lower_bound
 
     def check(self, value):
-        return self.lower_bound <= value
+        return self.lower_bound < value
 
 
 class _LessThan(Constraint):
     """
-    Constrain to a real half line `[inf, upper_bound]`.
+    Constrain to a real half line `[-inf, upper_bound)`.
     """
     def __init__(self, upper_bound):
         self.upper_bound = upper_bound
 
     def check(self, value):
-        return value <= self.upper_bound
+        return value < self.upper_bound
 
 
 class _Interval(Constraint):
@@ -177,7 +177,7 @@ class _Simplex(Constraint):
     Specifically: `x >= 0` and `x.sum(-1) == 1`.
     """
     def check(self, value):
-        return (value >= 0) & ((value.sum(-1, True) - 1).abs() < 1e-6)
+        return (value >= 0).all() & ((value.sum(-1, True) - 1).abs() < 1e-6).all()
 
 
 class _LowerTriangular(Constraint):
diff --git a/torch/distributions/dirichlet.py b/torch/distributions/dirichlet.py
index 0f32ad77e9eb..9058960652e5 100644
--- a/torch/distributions/dirichlet.py
+++ b/torch/distributions/dirichlet.py
@@ -56,10 +56,10 @@ class Dirichlet(ExponentialFamily):
     support = constraints.simplex
     has_rsample = True
 
-    def __init__(self, concentration):
+    def __init__(self, concentration, validate_args=None):
         self.concentration, = broadcast_all(concentration)
         batch_shape, event_shape = concentration.shape[:-1], concentration.shape[-1:]
-        super(Dirichlet, self).__init__(batch_shape, event_shape)
+        super(Dirichlet, self).__init__(batch_shape, event_shape, validate_args=validate_args)
 
     def rsample(self, sample_shape=()):
         shape = self._extended_shape(sample_shape)
@@ -69,7 +69,8 @@ def rsample(self, sample_shape=()):
         return _dirichlet_sample_nograd(concentration)
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return ((torch.log(value) * (self.concentration - 1.0)).sum(-1) +
                 torch.lgamma(self.concentration.sum(-1)) -
                 torch.lgamma(self.concentration).sum(-1))
diff --git a/torch/distributions/distribution.py b/torch/distributions/distribution.py
index bfeb935624b1..8903b3afb45f 100644
--- a/torch/distributions/distribution.py
+++ b/torch/distributions/distribution.py
@@ -1,6 +1,7 @@
 import torch
 from torch.autograd import Variable
 import warnings
+from torch.distributions import constraints
 
 
 class Distribution(object):
@@ -10,10 +11,27 @@ class Distribution(object):
 
     has_rsample = False
     has_enumerate_support = False
+    _validate_args = False
+    support = None
+    params = {}
 
-    def __init__(self, batch_shape=torch.Size(), event_shape=torch.Size()):
+    @staticmethod
+    def set_default_validate_args(value):
+        if value not in [True, False]:
+            raise ValueError
+        Distribution._validate_args = value
+
+    def __init__(self, batch_shape=torch.Size(), event_shape=torch.Size(), validate_args=None):
         self._batch_shape = batch_shape
         self._event_shape = event_shape
+        if validate_args is not None:
+            self._validate_args = validate_args
+        if self._validate_args:
+            if not constraints.is_dependent(self.params):
+                for param, constraint in self.params.items():
+                    if not constraints.is_dependent(constraint):
+                        if not constraint.check(self.__getattribute__(param)).all():
+                            raise ValueError("The parameter {} has invalid values".format(param))
 
     @property
     def batch_shape(self):
@@ -158,11 +176,12 @@ def _extended_shape(self, sample_shape=torch.Size()):
         """
         return torch.Size(sample_shape + self._batch_shape + self._event_shape)
 
-    def _validate_log_prob_arg(self, value):
+    def _validate_sample(self, value):
         """
-        Argument validation for `log_prob` methods. The rightmost dimensions
-        of a value to be scored via `log_prob` must agree with the distribution's
-        batch and event shapes.
+        Argument validation for distribution methods such as `log_prob`,
+        `cdf` and `icdf`. The rightmost dimensions of a value to be
+        scored via these methods must agree with the distribution's batch
+        and event shapes.
 
         Args:
             value (Tensor): the tensor whose log probability is to be
@@ -186,5 +205,8 @@ def _validate_log_prob_arg(self, value):
                 raise ValueError('Value is not broadcastable with batch_shape+event_shape: {} vs {}.'.
                                  format(actual_shape, expected_shape))
 
+        if not self.support.check(value).all():
+            raise ValueError('The value argument must be within the support')
+
     def __repr__(self):
         return self.__class__.__name__ + '()'
diff --git a/torch/distributions/exponential.py b/torch/distributions/exponential.py
index 9dfe80aeef1b..3ea2264c28fe 100644
--- a/torch/distributions/exponential.py
+++ b/torch/distributions/exponential.py
@@ -37,25 +37,28 @@ def stddev(self):
     def variance(self):
         return self.rate.pow(-2)
 
-    def __init__(self, rate):
+    def __init__(self, rate, validate_args=None):
         self.rate, = broadcast_all(rate)
         batch_shape = torch.Size() if isinstance(rate, Number) else self.rate.size()
-        super(Exponential, self).__init__(batch_shape)
+        super(Exponential, self).__init__(batch_shape, validate_args=validate_args)
 
     def rsample(self, sample_shape=torch.Size()):
         shape = self._extended_shape(sample_shape)
         return self.rate.new(shape).exponential_() / self.rate
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return self.rate.log() - self.rate * value
 
     def cdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return 1 - torch.exp(-self.rate * value)
 
     def icdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return -torch.log(1 - value) / self.rate
 
     def entropy(self):
diff --git a/torch/distributions/fishersnedecor.py b/torch/distributions/fishersnedecor.py
index d4e5dbacb565..f9fbb97505a1 100644
--- a/torch/distributions/fishersnedecor.py
+++ b/torch/distributions/fishersnedecor.py
@@ -26,7 +26,7 @@ class FisherSnedecor(Distribution):
     support = constraints.positive
     has_rsample = True
 
-    def __init__(self, df1, df2):
+    def __init__(self, df1, df2, validate_args=None):
         self.df1, self.df2 = broadcast_all(df1, df2)
         self._gamma1 = Gamma(self.df1 * 0.5, self.df1)
         self._gamma2 = Gamma(self.df2 * 0.5, self.df2)
@@ -35,7 +35,7 @@ def __init__(self, df1, df2):
             batch_shape = torch.Size()
         else:
             batch_shape = self.df1.size()
-        super(FisherSnedecor, self).__init__(batch_shape)
+        super(FisherSnedecor, self).__init__(batch_shape, validate_args=validate_args)
 
     @property
     def mean(self):
@@ -61,7 +61,8 @@ def rsample(self, sample_shape=torch.Size(())):
         return Y
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         ct1 = self.df1 * 0.5
         ct2 = self.df2 * 0.5
         ct3 = self.df1 / self.df2
diff --git a/torch/distributions/gamma.py b/torch/distributions/gamma.py
index 9f06fbb020a9..ed4d39602a81 100644
--- a/torch/distributions/gamma.py
+++ b/torch/distributions/gamma.py
@@ -42,13 +42,13 @@ def mean(self):
     def variance(self):
         return self.concentration / self.rate.pow(2)
 
-    def __init__(self, concentration, rate):
+    def __init__(self, concentration, rate, validate_args=None):
         self.concentration, self.rate = broadcast_all(concentration, rate)
         if isinstance(concentration, Number) and isinstance(rate, Number):
             batch_shape = torch.Size()
         else:
             batch_shape = self.concentration.size()
-        super(Gamma, self).__init__(batch_shape)
+        super(Gamma, self).__init__(batch_shape, validate_args=validate_args)
 
     def rsample(self, sample_shape=torch.Size()):
         shape = self._extended_shape(sample_shape)
@@ -58,7 +58,8 @@ def rsample(self, sample_shape=torch.Size()):
         return value
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return (self.concentration * torch.log(self.rate) +
                 (self.concentration - 1) * torch.log(value) -
                 self.rate * value - torch.lgamma(self.concentration))
diff --git a/torch/distributions/geometric.py b/torch/distributions/geometric.py
index fab91a43f272..73e18851515d 100644
--- a/torch/distributions/geometric.py
+++ b/torch/distributions/geometric.py
@@ -29,7 +29,7 @@ class Geometric(Distribution):
     params = {'probs': constraints.unit_interval}
     support = constraints.nonnegative_integer
 
-    def __init__(self, probs=None, logits=None):
+    def __init__(self, probs=None, logits=None, validate_args=None):
         if (probs is None) == (logits is None):
             raise ValueError("Either `probs` or `logits` must be specified, but not both.")
         if probs is not None:
@@ -43,7 +43,7 @@ def __init__(self, probs=None, logits=None):
             batch_shape = torch.Size()
         else:
             batch_shape = probs_or_logits.size()
-        super(Geometric, self).__init__(batch_shape)
+        super(Geometric, self).__init__(batch_shape, validate_args=validate_args)
 
     @property
     def mean(self):
@@ -68,7 +68,8 @@ def sample(self, sample_shape=torch.Size()):
             return (u.log() / (-self.probs).log1p()).floor()
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         value, probs = broadcast_all(value, self.probs.clone())
         probs[(probs == 1) & (value == 0)] = 0
         return value * (-probs).log1p() + self.probs.log()
diff --git a/torch/distributions/gumbel.py b/torch/distributions/gumbel.py
index fa5c08bb0f09..6ae38761bf9d 100644
--- a/torch/distributions/gumbel.py
+++ b/torch/distributions/gumbel.py
@@ -28,7 +28,7 @@ class Gumbel(TransformedDistribution):
     params = {'loc': constraints.real, 'scale': constraints.positive}
     support = constraints.real
 
-    def __init__(self, loc, scale):
+    def __init__(self, loc, scale, validate_args=None):
         self.loc, self.scale = broadcast_all(loc, scale)
         finfo = _finfo(self.loc)
         if isinstance(loc, Number) and isinstance(scale, Number):
@@ -39,7 +39,7 @@ def __init__(self, loc, scale):
             base_dist = Uniform(self.loc.new(self.loc.size()).fill_(finfo.tiny), 1 - finfo.eps)
         transforms = [ExpTransform().inv, AffineTransform(loc=0, scale=-torch.ones_like(self.scale)),
                       ExpTransform().inv, AffineTransform(loc=loc, scale=-self.scale)]
-        super(Gumbel, self).__init__(base_dist, transforms)
+        super(Gumbel, self).__init__(base_dist, transforms, validate_args=validate_args)
 
     @property
     def mean(self):
diff --git a/torch/distributions/laplace.py b/torch/distributions/laplace.py
index d4f615675407..5375a96ed5ee 100644
--- a/torch/distributions/laplace.py
+++ b/torch/distributions/laplace.py
@@ -36,13 +36,13 @@ def variance(self):
     def stddev(self):
         return (2 ** 0.5) * self.scale
 
-    def __init__(self, loc, scale):
+    def __init__(self, loc, scale, validate_args=None):
         self.loc, self.scale = broadcast_all(loc, scale)
         if isinstance(loc, Number) and isinstance(scale, Number):
             batch_shape = torch.Size()
         else:
             batch_shape = self.loc.size()
-        super(Laplace, self).__init__(batch_shape)
+        super(Laplace, self).__init__(batch_shape, validate_args=validate_args)
 
     def rsample(self, sample_shape=torch.Size()):
         shape = self._extended_shape(sample_shape)
@@ -52,15 +52,18 @@ def rsample(self, sample_shape=torch.Size()):
         return self.loc - self.scale * u.sign() * torch.log1p(-u.abs())
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return -torch.log(2 * self.scale) - torch.abs(value - self.loc) / self.scale
 
     def cdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return 0.5 - 0.5 * (value - self.loc).sign() * torch.expm1(-(value - self.loc).abs() / self.scale)
 
     def icdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         term = value - 0.5
         return self.loc - self.scale * (term).sign() * torch.log1p(-2 * term.abs())
 
diff --git a/torch/distributions/log_normal.py b/torch/distributions/log_normal.py
index e0e1b6b505f3..8d7880928a14 100644
--- a/torch/distributions/log_normal.py
+++ b/torch/distributions/log_normal.py
@@ -27,8 +27,8 @@ class LogNormal(TransformedDistribution):
     support = constraints.positive
     has_rsample = True
 
-    def __init__(self, loc, scale):
-        super(LogNormal, self).__init__(Normal(loc, scale), ExpTransform())
+    def __init__(self, loc, scale, validate_args=None):
+        super(LogNormal, self).__init__(Normal(loc, scale), ExpTransform(), validate_args=validate_args)
 
     @property
     def loc(self):
diff --git a/torch/distributions/multinomial.py b/torch/distributions/multinomial.py
index 3a277aa6aa4d..ebb3d5290cd0 100644
--- a/torch/distributions/multinomial.py
+++ b/torch/distributions/multinomial.py
@@ -52,14 +52,14 @@ def mean(self):
     def variance(self):
         return self.total_count * self.probs * (1 - self.probs)
 
-    def __init__(self, total_count=1, probs=None, logits=None):
+    def __init__(self, total_count=1, probs=None, logits=None, validate_args=None):
         if not isinstance(total_count, Number):
             raise NotImplementedError('inhomogeneous total_count is not supported')
         self.total_count = total_count
         self._categorical = Categorical(probs=probs, logits=logits)
         batch_shape = self._categorical.batch_shape
         event_shape = self._categorical.param_shape[-1:]
-        super(Multinomial, self).__init__(batch_shape, event_shape)
+        super(Multinomial, self).__init__(batch_shape, event_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._categorical._new(*args, **kwargs)
@@ -93,7 +93,8 @@ def sample(self, sample_shape=torch.Size()):
         return counts.type_as(self.probs)
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         logits, value = broadcast_all(self.logits.clone(), value)
         log_factorial_n = torch.lgamma(value.sum(-1) + 1)
         log_factorial_xs = torch.lgamma(value + 1).sum(-1)
diff --git a/torch/distributions/normal.py b/torch/distributions/normal.py
index 5ab606bf1040..6b0b44bc2834 100644
--- a/torch/distributions/normal.py
+++ b/torch/distributions/normal.py
@@ -41,13 +41,13 @@ def stddev(self):
     def variance(self):
         return self.stddev.pow(2)
 
-    def __init__(self, loc, scale):
+    def __init__(self, loc, scale, validate_args=None):
         self.loc, self.scale = broadcast_all(loc, scale)
         if isinstance(loc, Number) and isinstance(scale, Number):
             batch_shape = torch.Size()
         else:
             batch_shape = self.loc.size()
-        super(Normal, self).__init__(batch_shape)
+        super(Normal, self).__init__(batch_shape, validate_args=validate_args)
 
     def sample(self, sample_shape=torch.Size()):
         shape = self._extended_shape(sample_shape)
@@ -60,18 +60,21 @@ def rsample(self, sample_shape=torch.Size()):
         return self.loc + eps * self.scale
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         # compute the variance
         var = (self.scale ** 2)
         log_scale = math.log(self.scale) if isinstance(self.scale, Number) else self.scale.log()
         return -((value - self.loc) ** 2) / (2 * var) - log_scale - math.log(math.sqrt(2 * math.pi))
 
     def cdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return 0.5 * (1 + torch.erf((value - self.loc) * self.scale.reciprocal() / math.sqrt(2)))
 
     def icdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         return self.loc + self.scale * torch.erfinv(2 * value - 1) * math.sqrt(2)
 
     def entropy(self):
diff --git a/torch/distributions/one_hot_categorical.py b/torch/distributions/one_hot_categorical.py
index ab897c703700..440d1b9687cf 100644
--- a/torch/distributions/one_hot_categorical.py
+++ b/torch/distributions/one_hot_categorical.py
@@ -35,11 +35,11 @@ class OneHotCategorical(Distribution):
     support = constraints.simplex
     has_enumerate_support = True
 
-    def __init__(self, probs=None, logits=None):
+    def __init__(self, probs=None, logits=None, validate_args=None):
         self._categorical = Categorical(probs, logits)
         batch_shape = self._categorical.batch_shape
         event_shape = self._categorical.param_shape[-1:]
-        super(OneHotCategorical, self).__init__(batch_shape, event_shape)
+        super(OneHotCategorical, self).__init__(batch_shape, event_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._categorical._new(*args, **kwargs)
@@ -74,7 +74,8 @@ def sample(self, sample_shape=torch.Size()):
         return one_hot.scatter_(-1, indices, 1)
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         indices = value.max(-1)[1]
         return self._categorical.log_prob(indices)
 
diff --git a/torch/distributions/pareto.py b/torch/distributions/pareto.py
index cc37b80633ef..78c3d9afacb0 100644
--- a/torch/distributions/pareto.py
+++ b/torch/distributions/pareto.py
@@ -27,11 +27,11 @@ class Pareto(TransformedDistribution):
     """
     params = {'alpha': constraints.positive, 'scale': constraints.positive}
 
-    def __init__(self, scale, alpha):
+    def __init__(self, scale, alpha, validate_args=None):
         self.scale, self.alpha = broadcast_all(scale, alpha)
         base_dist = Exponential(self.alpha)
         transforms = [ExpTransform(), AffineTransform(loc=0, scale=self.scale)]
-        super(Pareto, self).__init__(base_dist, transforms)
+        super(Pareto, self).__init__(base_dist, transforms, validate_args=validate_args)
 
     @property
     def mean(self):
diff --git a/torch/distributions/poisson.py b/torch/distributions/poisson.py
index e9c5ec9d75ae..d50b600d019c 100644
--- a/torch/distributions/poisson.py
+++ b/torch/distributions/poisson.py
@@ -35,13 +35,13 @@ def mean(self):
     def variance(self):
         return self.rate
 
-    def __init__(self, rate):
+    def __init__(self, rate, validate_args=None):
         self.rate, = broadcast_all(rate)
         if isinstance(rate, Number):
             batch_shape = torch.Size()
         else:
             batch_shape = self.rate.size()
-        super(Poisson, self).__init__(batch_shape)
+        super(Poisson, self).__init__(batch_shape, validate_args=validate_args)
 
     def sample(self, sample_shape=torch.Size()):
         shape = self._extended_shape(sample_shape)
@@ -49,7 +49,8 @@ def sample(self, sample_shape=torch.Size()):
             return torch.poisson(self.rate.expand(shape))
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         rate, value = broadcast_all(self.rate, value)
         return (rate.log() * value) - rate - (value + 1).lgamma()
 
diff --git a/torch/distributions/relaxed_bernoulli.py b/torch/distributions/relaxed_bernoulli.py
index 3dedd4e9912b..b7f830893896 100644
--- a/torch/distributions/relaxed_bernoulli.py
+++ b/torch/distributions/relaxed_bernoulli.py
@@ -29,7 +29,7 @@ class LogitRelaxedBernoulli(Distribution):
     params = {'probs': constraints.unit_interval}
     support = constraints.real
 
-    def __init__(self, temperature, probs=None, logits=None):
+    def __init__(self, temperature, probs=None, logits=None, validate_args=None):
         self.temperature = temperature
         if (probs is None) == (logits is None):
             raise ValueError("Either `probs` or `logits` must be specified, but not both.")
@@ -44,7 +44,7 @@ def __init__(self, temperature, probs=None, logits=None):
             batch_shape = torch.Size()
         else:
             batch_shape = self._param.size()
-        super(LogitRelaxedBernoulli, self).__init__(batch_shape)
+        super(LogitRelaxedBernoulli, self).__init__(batch_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._param.new(*args, **kwargs)
@@ -68,7 +68,8 @@ def rsample(self, sample_shape=torch.Size()):
         return (uniforms.log() - (-uniforms).log1p() + probs.log() - (-probs).log1p()) / self.temperature
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         logits, value = broadcast_all(self.logits, value)
         diff = logits - value.mul(self.temperature)
         return self.temperature.log() + diff - 2 * diff.exp().log1p()
@@ -100,9 +101,9 @@ class RelaxedBernoulli(TransformedDistribution):
     support = constraints.unit_interval
     has_rsample = True
 
-    def __init__(self, temperature, probs=None, logits=None):
+    def __init__(self, temperature, probs=None, logits=None, validate_args=None):
         super(RelaxedBernoulli, self).__init__(LogitRelaxedBernoulli(temperature, probs, logits),
-                                               SigmoidTransform())
+                                               SigmoidTransform(), validate_args=validate_args)
 
     @property
     def temperature(self):
diff --git a/torch/distributions/relaxed_categorical.py b/torch/distributions/relaxed_categorical.py
index c541f8630bbb..9d1fc3e91247 100644
--- a/torch/distributions/relaxed_categorical.py
+++ b/torch/distributions/relaxed_categorical.py
@@ -32,12 +32,12 @@ class ExpRelaxedCategorical(Distribution):
     support = constraints.real
     has_rsample = True
 
-    def __init__(self, temperature, probs=None, logits=None):
+    def __init__(self, temperature, probs=None, logits=None, validate_args=None):
         self._categorical = Categorical(probs, logits)
         self.temperature = temperature
         batch_shape = self._categorical.batch_shape
         event_shape = self._categorical.param_shape[-1:]
-        super(ExpRelaxedCategorical, self).__init__(batch_shape, event_shape)
+        super(ExpRelaxedCategorical, self).__init__(batch_shape, event_shape, validate_args=validate_args)
 
     def _new(self, *args, **kwargs):
         return self._categorical._new(*args, **kwargs)
@@ -63,7 +63,8 @@ def rsample(self, sample_shape=torch.Size()):
 
     def log_prob(self, value):
         K = self._categorical._num_events
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         logits, value = broadcast_all(self.logits, value)
         log_scale = (self.temperature.new(self.temperature.shape).fill_(K).lgamma() -
                      self.temperature.log().mul(-(K - 1)))
@@ -98,9 +99,9 @@ class RelaxedOneHotCategorical(TransformedDistribution):
     support = constraints.simplex
     has_rsample = True
 
-    def __init__(self, temperature, probs=None, logits=None):
+    def __init__(self, temperature, probs=None, logits=None, validate_args=None):
         super(RelaxedOneHotCategorical, self).__init__(ExpRelaxedCategorical(temperature, probs, logits),
-                                                       ExpTransform())
+                                                       ExpTransform(), validate_args=validate_args)
 
     @property
     def temperature(self):
diff --git a/torch/distributions/studentT.py b/torch/distributions/studentT.py
index 02269b0e8d51..a94aff9d84c2 100644
--- a/torch/distributions/studentT.py
+++ b/torch/distributions/studentT.py
@@ -39,11 +39,11 @@ def variance(self):
         m[self.df <= 1] = float('nan')
         return m
 
-    def __init__(self, df, loc=0., scale=1.):
+    def __init__(self, df, loc=0., scale=1., validate_args=None):
         self.df, self.loc, self.scale = broadcast_all(df, loc, scale)
         self._chi2 = Chi2(df)
         batch_shape = torch.Size() if isinstance(df, Number) else self.df.size()
-        super(StudentT, self).__init__(batch_shape)
+        super(StudentT, self).__init__(batch_shape, validate_args=validate_args)
 
     def rsample(self, sample_shape=torch.Size()):
         # NOTE: This does not agree with scipy implementation as much as other distributions.
@@ -60,7 +60,8 @@ def rsample(self, sample_shape=torch.Size()):
         return self.loc + self.scale * Y
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         y = (value - self.loc) / self.scale
         Z = (self.scale.log() +
              0.5 * self.df.log() +
diff --git a/torch/distributions/transformed_distribution.py b/torch/distributions/transformed_distribution.py
index b4bcfdfc9165..dad8e1f34ddc 100644
--- a/torch/distributions/transformed_distribution.py
+++ b/torch/distributions/transformed_distribution.py
@@ -18,7 +18,7 @@ class TransformedDistribution(Distribution):
     maximum shape of its base distribution and its transforms, since transforms
     can introduce correlations among events.
     """
-    def __init__(self, base_distribution, transforms):
+    def __init__(self, base_distribution, transforms, validate_args=None):
         self.base_dist = base_distribution
         if isinstance(transforms, Transform):
             self.transforms = [transforms, ]
@@ -32,11 +32,9 @@ def __init__(self, base_distribution, transforms):
         event_dim = max([len(self.base_dist.event_shape)] + [t.event_dim for t in self.transforms])
         batch_shape = shape[:len(shape) - event_dim]
         event_shape = shape[len(shape) - event_dim:]
-        super(TransformedDistribution, self).__init__(batch_shape, event_shape)
+        super(TransformedDistribution, self).__init__(batch_shape, event_shape, validate_args=validate_args)
 
-    @constraints.dependent_property
-    def params(self):
-        return self.base_dist.params  # TODO add params of transforms?
+    params = {}
 
     @constraints.dependent_property
     def support(self):
@@ -76,7 +74,6 @@ def log_prob(self, value):
         Scores the sample by inverting the transform(s) and computing the score
         using the score of the base distribution and the log abs det jacobian.
         """
-        self.base_dist._validate_log_prob_arg(value)
         event_dim = len(self.event_shape)
         log_prob = 0.0
         y = value
@@ -85,6 +82,7 @@ def log_prob(self, value):
             log_prob -= _sum_rightmost(transform.log_abs_det_jacobian(x, y),
                                        event_dim - transform.event_dim)
             y = x
+
         log_prob += _sum_rightmost(self.base_dist.log_prob(y),
                                    event_dim - len(self.base_dist.event_shape))
         return log_prob
@@ -108,6 +106,8 @@ def cdf(self, value):
         """
         for transform in self.transforms[::-1]:
             value = transform.inv(value)
+        if self._validate_args:
+            self.base_dist._validate_sample(value)
         value = self.base_dist.cdf(value)
         value = self._monotonize_cdf(value)
         return value
@@ -118,6 +118,8 @@ def icdf(self, value):
         transform(s) and computing the score of the base distribution.
         """
         value = self._monotonize_cdf(value)
+        if self._validate_args:
+            self.base_dist._validate_sample(value)
         value = self.base_dist.icdf(value)
         for transform in self.transforms:
             value = transform(value)
diff --git a/torch/distributions/uniform.py b/torch/distributions/uniform.py
index effd5c0cdfd9..046ea42f188d 100644
--- a/torch/distributions/uniform.py
+++ b/torch/distributions/uniform.py
@@ -40,13 +40,17 @@ def stddev(self):
     def variance(self):
         return (self.high - self.low).pow(2) / 12
 
-    def __init__(self, low, high):
+    def __init__(self, low, high, validate_args=None):
         self.low, self.high = broadcast_all(low, high)
+
         if isinstance(low, Number) and isinstance(high, Number):
             batch_shape = torch.Size()
         else:
             batch_shape = self.low.size()
-        super(Uniform, self).__init__(batch_shape)
+        super(Uniform, self).__init__(batch_shape, validate_args=validate_args)
+
+        if self._validate_args and not torch.lt(self.low, self.high).all():
+            raise ValueError("Uniform is not defined when low>= high")
 
     @constraints.dependent_property
     def support(self):
@@ -58,18 +62,21 @@ def rsample(self, sample_shape=torch.Size()):
         return self.low + rand * (self.high - self.low)
 
     def log_prob(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         lb = value.ge(self.low).type_as(self.low)
         ub = value.lt(self.high).type_as(self.low)
         return torch.log(lb.mul(ub)) - torch.log(self.high - self.low)
 
     def cdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         result = (value - self.low) / (self.high - self.low)
         return result
 
     def icdf(self, value):
-        self._validate_log_prob_arg(value)
+        if self._validate_args:
+            self._validate_sample(value)
         result = value * (self.high - self.low) + self.low
         return result
 


Code review comments:
diff_hunk: @@ -32,11 +32,9 @@ def __init__(self, base_distribution, transforms):
         event_dim = max([len(self.base_dist.event_shape)] + [t.event_dim for t in self.transforms])
         batch_shape = shape[:len(shape) - event_dim]
         event_shape = shape[len(shape) - event_dim:]
-        super(TransformedDistribution, self).__init__(batch_shape, event_shape)
+        super(TransformedDistribution, self).__init__(batch_shape, event_shape, validate_args=validate_args)
 
-    @constraints.dependent_property
-    def params(self):
-        return self.base_dist.params  # TODO add params of transforms?
+    params = {}
path: torch/distributions/transformed_distribution.py
text: @user1:
Why do we have this in here?

@user2:
@author is fixing a type error here (my fault). `.params` should a dict mapping strings to `Constraint` objects, where the strings are kwargs keys of the distribution's `__init__()` method. However none of the arguments to `TransformedDistribution.__init__()` are tensors (the first arg is a `Distribution` and the second is a list of `Transform`s), hence none of the args can be constrained by `Constraint` objects, and hence `.params` should be empty.

It's fine for subclasses of `TransformedDistribution` to define a tensor-parameterized `__init__()` method and corresponding entries in `.params`, e.g. as we do with `LogNormal`.

@user1:
Yeah, but why can't we just return the parameters of the base distribution?

@author:
@user1 No, that does work as then it starts looking for the subclass parameters in the `TransformedDistribution` class which fails.

@user1:
Sorry, I didn't get that. What starts looking for the parameters?

@author:
`TestValidation.test_valid` test fails with the following error:


```
self = <test_distributions.TestValidation testMethod=test_valid>

    def test_valid(self):
        for Dist, params in EXAMPLES:
            for i, param in enumerate(params):
>               Dist(validate_args=True, **param)

test/test_distributions.py:3065: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
torch/distributions/transformed_distribution.py:35: in __init__
    super(TransformedDistribution, self).__init__(batch_shape, event_shape, validate_args=validate_args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = TransformedDistribution(), batch_shape = torch.Size([2, 3]), event_shape = torch.Size([]), validate_args = True

    def __init__(self, batch_shape=torch.Size(), event_shape=torch.Size(), validate_args=None):
        self._batch_shape = batch_shape
        self._event_shape = event_shape
        if validate_args is not None:
            self._validate_args = validate_args
        if self._validate_args:
            if not constraints.is_dependent(self.params):
                for param, constraint in self.params.items():
                    if not constraints.is_dependent(constraint):
>                       if not constraint.check(self.__getattribute__(param)).all():
E                       AttributeError: 'TransformedDistribution' object has no attribute 'loc'

torch/distributions/distribution.py:33: AttributeError

```

This is what i would have in the code:
```
class TransformedDistribution(Distribution):
  ....

    @user3.dependent_property
    def params(self):
        return self.base_dist.params
```

What do u suggest?

@user2:
@user1 To support generic programming, the `d.param` property should refer only to args of that distribution's `d.__init__()` method. If you want to examine the parameters of the base distribution of a transformed distribution, then you should call `my_transformed_dist.base_dist.params`. Note that most instances of transformed distributions are parameterized by not only their `base_dist` parameters but also parameters of their transforms, and we have no simple way to gather all those parameters automatically.

@author:
Hey guys, do the changes look fine? Please let me know

@user1:
So it's not so much about having the parameters be the `__init__` arguments, but it's more about them being attributes of the distributed later, correct? We could add `__getattr__` for `TransformedDistribution` such that it allows access to `base_dist` params. Alternatively we can do this, but it's a bit weird that we're lying about the parameter set.

@user2:
@user1 What do you think of declaring the non-tensor args as `dependent`?
```py
params = {'base_dist': constraints.dependent,
          'transforms': constraints.dependent}
```
@author does that still pass all your tests?

@author:
@user2 Yes. That passes the tests!

@user1:
hmm maybe I misunderstand what `params` are supposed to be. I thought it should be a mapping from names of attributes (that are tensors) to constraints imposed on them. This, however, is completely different. @user2 what are the semantics that you have in mind?

@user1:
NB I can merge it if it's blocking for you somehow (just let me know), but I wanted to clarify that. Right now the API seems inconsistent

@author:
@user2 let me know if you would like me to push this change. I want to merge this today.

@user2:
@user1 maybe I also misunderstand, sorry for any confusion. To avoid blocking @author why don't we try to revert this one change and you and I can discuss on Slack or Zoom or sth. 

@author apologies for confusion. Does it work if you revert this change and simply add a guard to the tests
```py
if constraints.is_dependent(my_class.params):
    continue
```

@author:
Done.

diff_hunk: @@ -2315,6 +2494,9 @@ def test_params_contains(self):
                         constraint = dist.params[name]
                     except KeyError:
                         continue  # ignore optional parameters
+                    except TypeError:
+                        continue
path: test/test_distributions.py
text: @user1:
This still needs to be addressed