Repository: GPflow/GPflow

Issue:
Positive transform fails large numbers
The positive transform implemented in `GPflow.transforms.positive` is an alias for the `Log1pe`-transform. The name suggests that it can be used to represent arbitrary positive numbers, however, it is actually only possible to to represent numbers up a value of about 700 due to the result of `np.exp` growing too large:

```python
In [31]: GPflow.transforms.positive.forward(700)
Out[31]: 700.000001

In [32]: GPflow.transforms.positive.forward(800)
/(...)/GPflow/transforms.py:129: RuntimeWarning: overflow encountered in exp
  return np.log(1. + np.exp(x)) + self._lower
Out[32]: inf

In [33]: GPflow.transforms.positive.backward(700)
Out[33]: array([ 699.999999])

In [34]: GPflow.transforms.positive.backward(800)
/(...)/GPflow/transforms.py:138: RuntimeWarning: overflow encountered in exp
  return np.log(np.exp(y - self._lower) - np.ones(1, np_float_type))
Out[34]: array([ inf])
```

Since the positive transform degrades to an identity function for large values, it should be fairly easy to fix this. However, it is probably important to ensure that both `forward` and `tf_forward` behave the same.

I guess a similar problem will exist for the `Logistic`-transform.


Discussion:
This is also highlighted in https://github.com/GPflow/GPflow/issues/300. I didn't think this was all too much of a problem, since values of 700 would be odd to encounter, but given the simplicity of the fix, it may be worth doing.
We should fix this, marking as a bug.

If the optimal value is out of range, then we'll *not* get an overflow during optimization, because the tensorflow representation of the forward transform won't overflow. But when the optimization finishes, we get the overflow when putting the optimized vector back into the model state. Then, a successive call to optimize will fail, because the reverse transform (to initialize the optimizer) will fail. 



tf.nn.softplus should do the trick:

https://www.tensorflow.org/api_docs/python/nn/activation_functions_#softplus

I'll have a look for a fix for logistic later I have a meeting with Zoubin this morning. 
We already use tf.nn.softplus on the tensorflow side, the problem comes when setting the values, at the end of optimization, which uses the numpy transform. I'll push a fix for Log1pe, which I think is important. I'll leave Logistic for you, @alexggmatthews :)

Resolving patch:
diff --git a/GPflow/transforms.py b/GPflow/transforms.py
--- a/GPflow/transforms.py
+++ b/GPflow/transforms.py
@@ -126,7 +126,9 @@ def __init__(self, lower=1e-6):
         self._lower = lower
 
     def forward(self, x):
-        return np.log(1. + np.exp(x)) + self._lower
+        result = np.log(1. + np.exp(x)) + self._lower
+        # do not transform large numbers, they overflow and the mapping is exactly identity.
+        return np.where(x > 35, x + self._lower, result)
 
     def tf_forward(self, x):
         return tf.nn.softplus(x) + self._lower
@@ -135,7 +137,8 @@ def tf_log_jacobian(self, x):
         return -tf.reduce_sum(tf.log(1. + tf.exp(-x)))
 
     def backward(self, y):
-        return np.log(np.exp(y - self._lower) - np.ones(1, np_float_type))
+        result = np.log(np.exp(y - self._lower) - np.ones(1, np_float_type))
+        return np.where(y > 35, y-self._lower, result)
 
     def __str__(self):
         return '+ve'


Test patch:
diff --git a/testing/test_transforms.py b/testing/test_transforms.py
--- a/testing/test_transforms.py
+++ b/testing/test_transforms.py
@@ -68,6 +68,20 @@ def jacobian(f):
                                         self.session.run(j2, feed_dict={self.x: self.x_np})))
 
 
+class TestOverflow(unittest.TestCase):
+    """
+    Bug #302 identified an overflow in the standard positive transform. This is a regression test.
+    """
+
+    def setUp(self):
+        self.t = GPflow.transforms.Log1pe()
+
+    def testOverflow(self):
+        y = self.t.forward(np.array([-300, -10, 10, 300]))
+        self.assertFalse(np.any(np.isinf(y)))
+        self.assertFalse(np.any(np.isnan(y)))
+
+
 class TestLowerTriTransform(unittest.TestCase):
     """
     Some extra tests for the LowerTriangle transformation.