Repository: BayesWitnesses/m2cgen (language: rust)

Issue:
sigmoid and softmax as language-specific functions
Maybe it is better to require from supported languages to implement `sigmoid` and `softmax` functions than defining them as expressions? It will simplify the readability of generated code and speed it up by more efficient native implementations.

Also, we can fallback to the current expressions when functions are missed.

Why I'm rising this issue is because I think that we currently have some kind of inconsistency, requiring implementation of `Tanh` function from languages, but at the same time defining `sigmoid` as expression, while they both can be written via `Exp` function.

![image](https://user-images.githubusercontent.com/25141164/79625245-57d1c500-8130-11ea-97d7-2b66d1a2a8b4.png)

![image](https://user-images.githubusercontent.com/25141164/79625258-69b36800-8130-11ea-97cb-8a640d2f9cb7.png)

https://github.com/BayesWitnesses/m2cgen/blob/dcd62f4476daf883b04cc40a85c125dd11245cd9/m2cgen/assemblers/utils.py#L83-L100

https://github.com/BayesWitnesses/m2cgen/blob/dcd62f4476daf883b04cc40a85c125dd11245cd9/m2cgen/interpreters/visual_basic/tanh.bas#L1-L11




Discussion / hints:
This is a great suggestion indeed. I agree with every point you raised. 

My only concern is that sigmoid and softmax are uncommon enough (unlike exponent and tanh) that we'll have to introduce a lot of manually written boilerplate across all languages. We already have some of it (vector multiplication for instance), but adding more will increase the support burden and reduce the ease of integration of new languages. 

Quite a few languages have been introduced by  external contributors and I'm afraid that by increasing complexity and raising the entry barrier we'll drive potential contributors away. 

Please let me know if you find my concerns valid.
I absolutely agree with you! As always, it is a tradeoff between complexity of adding languages and further runtime performance. However, I believe that the following my suggestion can eliminate potential  increase of entry point efforts.

> Also, we can fallback to the current expressions when functions are missed.

Moreover, it can even decrease the barrier. For example, if we had a default expression for `Tanh` function, there was no need to write custom implementation for the Dart language in #165.

To sum up, newcomers will have a choice either add very basic support of their favorite language without writing many math and statistical functions (however, I guess it won't be so hard for them as they should know that language well and we will have a lot of examples in other already supported languages) relying on our inefficient fallback expressions, or code a little bit more in their favorite language and make runtime performance better. Also, it will allow to break down adding support of new language into several PRs, while preserving the support of all ML models starting from the first one.

Speaking about the current language pool, I can write all needed functions for all of them in next days. And I have a wish to do my best helping in or creating "advanced-level support" PRs for new languages.

> My only concern is that sigmoid and softmax are uncommon enough

That's true, that not so many languages have them as a built-in functions. However, at present they are used only in boosting models and according to created issues in our GitHub repo it can be said that boosting models are the most popular ones. So, I believe it's worth to increase the support burden with the aim to increase the runtime performance for those models.
Oh shoot, I overlooked this statement, sorry.

> Also, we can fallback to the current expressions when functions are missed.

Though some time after posting my comment I came to the same conclusion myself. Yes, I think this idea is fantastic 👍 
@StrikerRUS one more question. Unless the implementation of those functions is provided by the standard library how exactly are we going to ensure:
> speed it up by more efficient native implementations

What will be different compared to the existing expression (with some reused/cached parts)?
It is hard to me to give a clear answer right now.

For instance, in `Tanh` we are clipping argument range with different thresholds.

Also, it will allow to decrease the depth of binary expressions and use `BinExpressionDepthTrackingMixin` less frequent.

Maybe for some functions loops will be more suitable, idk...

Fix patch:
diff --git a/.flake8 b/.flake8
index ed8c1220..8a36074b 100644
--- a/.flake8
+++ b/.flake8
@@ -5,3 +5,4 @@ exclude =
     generated_code_examples/,
     m2cgen/interpreters/python/linear_algebra.py,
     m2cgen/interpreters/python/softmax.py
+    m2cgen/interpreters/python/sigmoid.py
diff --git a/MANIFEST.in b/MANIFEST.in
index 06b0e1d3..c0a26428 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -5,4 +5,5 @@ recursive-include m2cgen log1p.*
 recursive-include m2cgen tanh.*
 recursive-include m2cgen atan.*
 recursive-include m2cgen softmax.*
+recursive-include m2cgen sigmoid.*
 global-exclude *.py[cod]
diff --git a/m2cgen/ast.py b/m2cgen/ast.py
index 87f090e8..e7f551e3 100644
--- a/m2cgen/ast.py
+++ b/m2cgen/ast.py
@@ -154,6 +154,23 @@ def __hash__(self):
         return hash(self.expr)
 
 
+class SigmoidExpr(NumExpr):
+    def __init__(self, expr, to_reuse=False):
+        assert expr.output_size == 1, "Only scalars are supported"
+
+        self.expr = expr
+        self.to_reuse = to_reuse
+
+    def __str__(self):
+        return f"SigmoidExpr({self.expr},to_reuse={self.to_reuse})"
+
+    def __eq__(self, other):
+        return type(other) is SigmoidExpr and self.expr == other.expr
+
+    def __hash__(self):
+        return hash(self.expr)
+
+
 class SqrtExpr(NumExpr):
     def __init__(self, expr, to_reuse=False):
         assert expr.output_size == 1, "Only scalars are supported"
@@ -411,7 +428,7 @@ def __hash__(self):
     ((VectorVal, SoftmaxExpr), lambda e: e.exprs),
     ((IfExpr), lambda e: [e.test, e.body, e.orelse]),
     ((AbsExpr, AtanExpr, ExpExpr, IdExpr, LogExpr, Log1pExpr,
-      SqrtExpr, TanhExpr),
+      SigmoidExpr, SqrtExpr, TanhExpr),
      lambda e: [e.expr]),
 ]
 
diff --git a/m2cgen/interpreters/c/interpreter.py b/m2cgen/interpreters/c/interpreter.py
index 09ecb987..65dc4854 100644
--- a/m2cgen/interpreters/c/interpreter.py
+++ b/m2cgen/interpreters/c/interpreter.py
@@ -23,10 +23,12 @@ class CInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "log"
     log1p_function_name = "log1p"
     power_function_name = "pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "sqrt"
     tanh_function_name = "tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -69,6 +71,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.c")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.c")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         if self.with_vectors:
             self._cg.add_dependency("<string.h>")
 
@@ -122,3 +128,7 @@ def interpret_softmax_expr(self, expr, **kwargs):
             var_name)
         self._cg.add_code_line(f"{func_inv};")
         return var_name
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/c/sigmoid.c b/m2cgen/interpreters/c/sigmoid.c
new file mode 100644
index 00000000..46d4fd9d
--- /dev/null
+++ b/m2cgen/interpreters/c/sigmoid.c
@@ -0,0 +1,7 @@
+double sigmoid(double x) {
+    if (x < 0.0) {
+        double z = exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + exp(-x));
+}
diff --git a/m2cgen/interpreters/c_sharp/interpreter.py b/m2cgen/interpreters/c_sharp/interpreter.py
index 64498bfc..de4fdf42 100644
--- a/m2cgen/interpreters/c_sharp/interpreter.py
+++ b/m2cgen/interpreters/c_sharp/interpreter.py
@@ -23,11 +23,13 @@ class CSharpInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "Log"
     log1p_function_name = "Log1p"
     power_function_name = "Pow"
+    sigmoid_function_name = "Sigmoid"
     softmax_function_name = "Softmax"
     sqrt_function_name = "Sqrt"
     tanh_function_name = "Tanh"
 
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, namespace="ML", class_name="Model", indent=4,
@@ -71,6 +73,10 @@ def interpret(self, expr):
                     filename = os.path.join(current_dir, "softmax.cs")
                     self._cg.add_code_lines(utils.get_file_content(filename))
 
+                if self.with_sigmoid_expr:
+                    filename = os.path.join(current_dir, "sigmoid.cs")
+                    self._cg.add_code_lines(utils.get_file_content(filename))
+
         if self.with_math_module:
             self._cg.add_dependency("System.Math")
 
@@ -83,3 +89,7 @@ def interpret_log1p_expr(self, expr, **kwargs):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/c_sharp/sigmoid.cs b/m2cgen/interpreters/c_sharp/sigmoid.cs
new file mode 100644
index 00000000..3dbdc4fd
--- /dev/null
+++ b/m2cgen/interpreters/c_sharp/sigmoid.cs
@@ -0,0 +1,7 @@
+private static double Sigmoid(double x) {
+    if (x < 0.0) {
+        double z = Exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + Exp(-x));
+}
diff --git a/m2cgen/interpreters/dart/interpreter.py b/m2cgen/interpreters/dart/interpreter.py
index 632c79a7..20420d3e 100644
--- a/m2cgen/interpreters/dart/interpreter.py
+++ b/m2cgen/interpreters/dart/interpreter.py
@@ -26,11 +26,13 @@ class DartInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "log"
     log1p_function_name = "log1p"
     power_function_name = "pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "sqrt"
     tanh_function_name = "tanh"
 
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
     with_tanh_expr = False
 
@@ -64,6 +66,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "log1p.dart")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.dart")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         if self.with_softmax_expr:
             filename = os.path.join(current_dir, "softmax.dart")
             self._cg.add_code_lines(utils.get_file_content(filename))
@@ -101,3 +107,7 @@ def interpret_tanh_expr(self, expr, **kwargs):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/dart/sigmoid.dart b/m2cgen/interpreters/dart/sigmoid.dart
new file mode 100644
index 00000000..46d4fd9d
--- /dev/null
+++ b/m2cgen/interpreters/dart/sigmoid.dart
@@ -0,0 +1,7 @@
+double sigmoid(double x) {
+    if (x < 0.0) {
+        double z = exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + exp(-x));
+}
diff --git a/m2cgen/interpreters/f_sharp/interpreter.py b/m2cgen/interpreters/f_sharp/interpreter.py
index 1ec46cd0..baa8b445 100644
--- a/m2cgen/interpreters/f_sharp/interpreter.py
+++ b/m2cgen/interpreters/f_sharp/interpreter.py
@@ -30,11 +30,13 @@ class FSharpInterpreter(FunctionalToCodeInterpreter,
     exponent_function_name = "exp"
     logarithm_function_name = "log"
     log1p_function_name = "log1p"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "sqrt"
     tanh_function_name = "tanh"
 
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -69,6 +71,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.fs")
             self._cg.prepend_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.fs")
+            self._cg.prepend_code_lines(utils.get_file_content(filename))
+
         return self._cg.finalize_and_get_generated_code()
 
     def create_code_generator(self):
@@ -88,6 +94,10 @@ def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
 
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
+
     def _dump_cache(self):
         if self._cached_expr_results:
             for func_name, expr_result in self._cached_expr_results.values():
diff --git a/m2cgen/interpreters/f_sharp/sigmoid.fs b/m2cgen/interpreters/f_sharp/sigmoid.fs
new file mode 100644
index 00000000..a7ff4602
--- /dev/null
+++ b/m2cgen/interpreters/f_sharp/sigmoid.fs
@@ -0,0 +1,5 @@
+let private sigmoid x =
+    let z = exp x
+    match x with
+        | i when i < 0.0 -> z / (1.0 + z)
+        | _ -> 1.0 / (1.0 + exp (-x))
diff --git a/m2cgen/interpreters/go/interpreter.py b/m2cgen/interpreters/go/interpreter.py
index a80a2b11..be048bf0 100644
--- a/m2cgen/interpreters/go/interpreter.py
+++ b/m2cgen/interpreters/go/interpreter.py
@@ -22,10 +22,12 @@ class GoInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "math.Log"
     log1p_function_name = "math.Log1p"
     power_function_name = "math.Pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "math.Sqrt"
     tanh_function_name = "math.Tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -59,6 +61,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.go")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.go")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         if self.with_math_module:
             self._cg.add_dependency("math")
 
@@ -67,3 +73,7 @@ def interpret(self, expr):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/go/sigmoid.go b/m2cgen/interpreters/go/sigmoid.go
new file mode 100644
index 00000000..6f5b144d
--- /dev/null
+++ b/m2cgen/interpreters/go/sigmoid.go
@@ -0,0 +1,7 @@
+func sigmoid(x float64) float64 {
+    if (x < 0.0) {
+        z := math.Exp(x)
+        return z / (1.0 + z)
+    }
+    return 1.0 / (1.0 + math.Exp(-x))
+}
diff --git a/m2cgen/interpreters/haskell/interpreter.py b/m2cgen/interpreters/haskell/interpreter.py
index 1fdcc368..87c5adc7 100644
--- a/m2cgen/interpreters/haskell/interpreter.py
+++ b/m2cgen/interpreters/haskell/interpreter.py
@@ -21,11 +21,13 @@ class HaskellInterpreter(FunctionalToCodeInterpreter,
     exponent_function_name = "exp"
     logarithm_function_name = "log"
     log1p_function_name = "log1p"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "sqrt"
     tanh_function_name = "tanh"
 
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self,  module_name="Model", indent=4, function_name="score",
@@ -65,6 +67,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.hs")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.hs")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         self._cg.prepend_code_line(self._cg.tpl_module_definition(
             module_name=self.module_name))
 
@@ -87,6 +93,10 @@ def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
 
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
+
     def _dump_cache(self):
         if self._cached_expr_results:
             self._cg.add_code_line("where")
diff --git a/m2cgen/interpreters/haskell/sigmoid.hs b/m2cgen/interpreters/haskell/sigmoid.hs
new file mode 100644
index 00000000..58bdc037
--- /dev/null
+++ b/m2cgen/interpreters/haskell/sigmoid.hs
@@ -0,0 +1,6 @@
+sigmoid :: Double -> Double
+sigmoid x
+    | x < 0.0 = z / (1.0 + z)
+    | otherwise = 1.0 / (1.0 + exp (-x))
+  where
+    z = exp x
diff --git a/m2cgen/interpreters/interpreter.py b/m2cgen/interpreters/interpreter.py
index ec329317..7af7ca56 100644
--- a/m2cgen/interpreters/interpreter.py
+++ b/m2cgen/interpreters/interpreter.py
@@ -87,6 +87,7 @@ class ToCodeInterpreter(BaseToCodeInterpreter):
     logarithm_function_name = NotImplemented
     log1p_function_name = NotImplemented
     power_function_name = NotImplemented
+    sigmoid_function_name = NotImplemented
     softmax_function_name = NotImplemented
     sqrt_function_name = NotImplemented
     tanh_function_name = NotImplemented
@@ -170,6 +171,15 @@ def interpret_log1p_expr(self, expr, **kwargs):
         return self._cg.function_invocation(
             self.log1p_function_name, nested_result)
 
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        if self.sigmoid_function_name is NotImplemented:
+            return self._do_interpret(
+                fallback_expressions.sigmoid(expr.expr), **kwargs)
+        self.with_math_module = True
+        nested_result = self._do_interpret(expr.expr, **kwargs)
+        return self._cg.function_invocation(
+            self.sigmoid_function_name, nested_result)
+
     def interpret_softmax_expr(self, expr, **kwargs):
         if self.softmax_function_name is NotImplemented:
             return self._do_interpret(
diff --git a/m2cgen/interpreters/java/interpreter.py b/m2cgen/interpreters/java/interpreter.py
index 18db49bc..a214e3d6 100644
--- a/m2cgen/interpreters/java/interpreter.py
+++ b/m2cgen/interpreters/java/interpreter.py
@@ -29,10 +29,12 @@ class JavaInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "Math.log"
     log1p_function_name = "Math.log1p"
     power_function_name = "Math.pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "Math.sqrt"
     tanh_function_name = "Math.tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, package_name=None, class_name="Model", indent=4,
@@ -71,6 +73,10 @@ def interpret(self, expr):
                 filename = os.path.join(current_dir, "softmax.java")
                 top_cg.add_code_lines(utils.get_file_content(filename))
 
+            if self.with_sigmoid_expr:
+                filename = os.path.join(current_dir, "sigmoid.java")
+                top_cg.add_code_lines(utils.get_file_content(filename))
+
         return top_cg.finalize_and_get_generated_code()
 
     # Required by SubroutinesMixin to create new code generator for
@@ -81,3 +87,7 @@ def create_code_generator(self):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/java/sigmoid.java b/m2cgen/interpreters/java/sigmoid.java
new file mode 100644
index 00000000..e8ec375b
--- /dev/null
+++ b/m2cgen/interpreters/java/sigmoid.java
@@ -0,0 +1,7 @@
+private static double sigmoid(double x) {
+    if (x < 0.0) {
+        double z = Math.exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + Math.exp(-x));
+}
diff --git a/m2cgen/interpreters/javascript/interpreter.py b/m2cgen/interpreters/javascript/interpreter.py
index a3f3139a..8a72f054 100644
--- a/m2cgen/interpreters/javascript/interpreter.py
+++ b/m2cgen/interpreters/javascript/interpreter.py
@@ -23,10 +23,12 @@ class JavascriptInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "Math.log"
     log1p_function_name = "Math.log1p"
     power_function_name = "Math.pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "Math.sqrt"
     tanh_function_name = "Math.tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score",
@@ -59,8 +61,16 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.js")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.js")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         return self._cg.finalize_and_get_generated_code()
 
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/javascript/sigmoid.js b/m2cgen/interpreters/javascript/sigmoid.js
new file mode 100644
index 00000000..c8c26ff8
--- /dev/null
+++ b/m2cgen/interpreters/javascript/sigmoid.js
@@ -0,0 +1,7 @@
+function sigmoid(x) {
+    if (x < 0.0) {
+        let z = Math.exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + Math.exp(-x));
+}
diff --git a/m2cgen/interpreters/php/interpreter.py b/m2cgen/interpreters/php/interpreter.py
index 816d6fb3..3352dd4f 100644
--- a/m2cgen/interpreters/php/interpreter.py
+++ b/m2cgen/interpreters/php/interpreter.py
@@ -23,10 +23,12 @@ class PhpInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "log"
     log1p_function_name = "log1p"
     power_function_name = "pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "sqrt"
     tanh_function_name = "tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -55,6 +57,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.php")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.php")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         self._cg.prepend_code_line("<?php")
 
         return self._cg.finalize_and_get_generated_code()
@@ -62,3 +68,7 @@ def interpret(self, expr):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/php/sigmoid.php b/m2cgen/interpreters/php/sigmoid.php
new file mode 100644
index 00000000..d48231d4
--- /dev/null
+++ b/m2cgen/interpreters/php/sigmoid.php
@@ -0,0 +1,7 @@
+function sigmoid($x) {
+    if ($x < 0.0) {
+        $z = exp($x);
+        return $z / (1.0 + $z);
+    }
+    return 1.0 / (1.0 + exp(-$x));
+}
diff --git a/m2cgen/interpreters/powershell/interpreter.py b/m2cgen/interpreters/powershell/interpreter.py
index 54ce78e9..03f4bdca 100644
--- a/m2cgen/interpreters/powershell/interpreter.py
+++ b/m2cgen/interpreters/powershell/interpreter.py
@@ -23,11 +23,13 @@ class PowershellInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "[math]::Log"
     log1p_function_name = "Log1p"
     power_function_name = "[math]::Pow"
+    sigmoid_function_name = "Sigmoid"
     softmax_function_name = "Softmax"
     sqrt_function_name = "[math]::Sqrt"
     tanh_function_name = "[math]::Tanh"
 
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="Score", *args, **kwargs):
@@ -62,6 +64,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.ps1")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.ps1")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         return self._cg.finalize_and_get_generated_code()
 
     def interpret_abs_expr(self, expr, **kwargs):
@@ -107,3 +113,7 @@ def interpret_pow_expr(self, expr, **kwargs):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/powershell/sigmoid.ps1 b/m2cgen/interpreters/powershell/sigmoid.ps1
new file mode 100644
index 00000000..a76dfce3
--- /dev/null
+++ b/m2cgen/interpreters/powershell/sigmoid.ps1
@@ -0,0 +1,7 @@
+function Sigmoid([double] $x) {
+    if ($x -lt 0.0) {
+        [double] $z = [math]::Exp($x)
+        return $z / (1.0 + $z)
+    }
+    return 1.0 / (1.0 + [math]::Exp(-$x))
+}
diff --git a/m2cgen/interpreters/python/interpreter.py b/m2cgen/interpreters/python/interpreter.py
index 06fed779..56b0cf8c 100644
--- a/m2cgen/interpreters/python/interpreter.py
+++ b/m2cgen/interpreters/python/interpreter.py
@@ -27,10 +27,12 @@ class PythonInterpreter(ImperativeToCodeInterpreter,
     logarithm_function_name = "math.log"
     log1p_function_name = "math.log1p"
     power_function_name = "math.pow"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "math.sqrt"
     tanh_function_name = "math.tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -59,6 +61,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.py")
             self._cg.prepend_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.py")
+            self._cg.prepend_code_lines(utils.get_file_content(filename))
+
         if self.with_math_module:
             self._cg.add_dependency("math")
 
@@ -72,3 +78,7 @@ def interpret_abs_expr(self, expr, **kwargs):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/python/sigmoid.py b/m2cgen/interpreters/python/sigmoid.py
new file mode 100644
index 00000000..4410f121
--- /dev/null
+++ b/m2cgen/interpreters/python/sigmoid.py
@@ -0,0 +1,5 @@
+def sigmoid(x):
+    if x < 0.0:
+        z = math.exp(x)
+        return z / (1.0 + z)
+    return 1.0 / (1.0 + math.exp(-x))
diff --git a/m2cgen/interpreters/r/interpreter.py b/m2cgen/interpreters/r/interpreter.py
index ce46514d..7679fe59 100644
--- a/m2cgen/interpreters/r/interpreter.py
+++ b/m2cgen/interpreters/r/interpreter.py
@@ -29,10 +29,12 @@ class RInterpreter(ImperativeToCodeInterpreter,
     exponent_function_name = "exp"
     logarithm_function_name = "log"
     log1p_function_name = "log1p"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "sqrt"
     tanh_function_name = "tanh"
 
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -53,6 +55,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.r")
             top_cg.prepend_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.r")
+            top_cg.prepend_code_lines(utils.get_file_content(filename))
+
         return top_cg.finalize_and_get_generated_code()
 
     def create_code_generator(self):
@@ -68,6 +74,10 @@ def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
 
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
+
     def interpret_bin_vector_expr(self, expr, **kwargs):
         self.with_linear_algebra = True
         return self._cg.infix_expression(
diff --git a/m2cgen/interpreters/r/sigmoid.r b/m2cgen/interpreters/r/sigmoid.r
new file mode 100644
index 00000000..603aa852
--- /dev/null
+++ b/m2cgen/interpreters/r/sigmoid.r
@@ -0,0 +1,7 @@
+sigmoid <- function(x) {
+    if (x < 0.0) {
+        z <- exp(x)
+        return(z / (1.0 + z))
+    }
+    return(1.0 / (1.0 + exp(-x)))
+}
diff --git a/m2cgen/interpreters/ruby/interpreter.py b/m2cgen/interpreters/ruby/interpreter.py
index 789a74fc..c5ff10de 100644
--- a/m2cgen/interpreters/ruby/interpreter.py
+++ b/m2cgen/interpreters/ruby/interpreter.py
@@ -22,11 +22,13 @@ class RubyInterpreter(ImperativeToCodeInterpreter,
     exponent_function_name = "Math.exp"
     logarithm_function_name = "Math.log"
     log1p_function_name = "log1p"
+    sigmoid_function_name = "sigmoid"
     softmax_function_name = "softmax"
     sqrt_function_name = "Math.sqrt"
     tanh_function_name = "Math.tanh"
 
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
 
     def __init__(self, indent=4, function_name="score", *args, **kwargs):
@@ -59,6 +61,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "softmax.rb")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.rb")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         return self._cg.finalize_and_get_generated_code()
 
     def interpret_bin_num_expr(self, expr, **kwargs):
@@ -90,3 +96,7 @@ def interpret_log1p_expr(self, expr, **kwargs):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/ruby/sigmoid.rb b/m2cgen/interpreters/ruby/sigmoid.rb
new file mode 100644
index 00000000..d6fd0829
--- /dev/null
+++ b/m2cgen/interpreters/ruby/sigmoid.rb
@@ -0,0 +1,7 @@
+def sigmoid(x)
+    if x < 0.0
+        z = Math.exp(x)
+        return z / (1.0 + z)
+    end
+    return 1.0 / (1.0 + Math.exp(-x))
+end
diff --git a/m2cgen/interpreters/visual_basic/interpreter.py b/m2cgen/interpreters/visual_basic/interpreter.py
index 08dc3d9f..463e7d88 100644
--- a/m2cgen/interpreters/visual_basic/interpreter.py
+++ b/m2cgen/interpreters/visual_basic/interpreter.py
@@ -21,11 +21,13 @@ class VisualBasicInterpreter(ImperativeToCodeInterpreter,
     exponent_function_name = "Math.Exp"
     logarithm_function_name = "Math.Log"
     log1p_function_name = "Log1p"
+    sigmoid_function_name = "Sigmoid"
     softmax_function_name = "Softmax"
     tanh_function_name = "Tanh"
 
     with_atan_expr = False
     with_log1p_expr = False
+    with_sigmoid_expr = False
     with_softmax_expr = False
     with_tanh_expr = False
 
@@ -65,6 +67,10 @@ def interpret(self, expr):
             filename = os.path.join(current_dir, "log1p.bas")
             self._cg.add_code_lines(utils.get_file_content(filename))
 
+        if self.with_sigmoid_expr:
+            filename = os.path.join(current_dir, "sigmoid.bas")
+            self._cg.add_code_lines(utils.get_file_content(filename))
+
         if self.with_softmax_expr:
             filename = os.path.join(current_dir, "softmax.bas")
             self._cg.add_code_lines(utils.get_file_content(filename))
@@ -102,3 +108,7 @@ def interpret_atan_expr(self, expr, **kwargs):
     def interpret_softmax_expr(self, expr, **kwargs):
         self.with_softmax_expr = True
         return super().interpret_softmax_expr(expr, **kwargs)
+
+    def interpret_sigmoid_expr(self, expr, **kwargs):
+        self.with_sigmoid_expr = True
+        return super().interpret_sigmoid_expr(expr, **kwargs)
diff --git a/m2cgen/interpreters/visual_basic/sigmoid.bas b/m2cgen/interpreters/visual_basic/sigmoid.bas
new file mode 100644
index 00000000..f9ff58a5
--- /dev/null
+++ b/m2cgen/interpreters/visual_basic/sigmoid.bas
@@ -0,0 +1,9 @@
+Function Sigmoid(ByVal number As Double) As Double
+    If number < 0.0 Then
+        Dim z As Double
+        z = Math.Exp(number)
+        Sigmoid = z / (1.0 + z)
+        Exit Function
+    End If
+    Sigmoid = 1.0 / (1.0 + Math.Exp(-number))
+End Function


Test patch:
diff --git a/tests/interpreters/test_c.py b/tests/interpreters/test_c.py
index 09f2a352..8b1eaeab 100644
--- a/tests/interpreters/test_c.py
+++ b/tests/interpreters/test_c.py
@@ -354,6 +354,27 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    interpreter = CInterpreter()
+
+    expected_code = """
+#include <math.h>
+double score(double * input) {
+    return sigmoid(2.0);
+}
+double sigmoid(double x) {
+    if (x < 0.0) {
+        double z = exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + exp(-x));
+}"""
+
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_c_sharp.py b/tests/interpreters/test_c_sharp.py
index b89672c6..6a8549f2 100644
--- a/tests/interpreters/test_c_sharp.py
+++ b/tests/interpreters/test_c_sharp.py
@@ -524,6 +524,31 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+using static System.Math;
+namespace ML {
+    public static class Model {
+        public static double Score(double[] input) {
+            return Sigmoid(2.0);
+        }
+        private static double Sigmoid(double x) {
+            if (x < 0.0) {
+                double z = Exp(x);
+                return z / (1.0 + z);
+            }
+            return 1.0 / (1.0 + Exp(-x));
+        }
+    }
+}
+"""
+
+    interpreter = CSharpInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_dart.py b/tests/interpreters/test_dart.py
index c8f7496f..694fc040 100644
--- a/tests/interpreters/test_dart.py
+++ b/tests/interpreters/test_dart.py
@@ -592,6 +592,26 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+import 'dart:math';
+double score(List<double> input) {
+    return sigmoid(2.0);
+}
+double sigmoid(double x) {
+    if (x < 0.0) {
+        double z = exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + exp(-x));
+}
+"""
+    interpreter = DartInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_f_sharp.py b/tests/interpreters/test_f_sharp.py
index b9fca010..66f88aba 100644
--- a/tests/interpreters/test_f_sharp.py
+++ b/tests/interpreters/test_f_sharp.py
@@ -472,6 +472,23 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+let private sigmoid x =
+    let z = exp x
+    match x with
+        | i when i < 0.0 -> z / (1.0 + z)
+        | _ -> 1.0 / (1.0 + exp (-x))
+let score (input : double list) =
+    sigmoid (2.0)
+"""
+
+    interpreter = FSharpInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_go.py b/tests/interpreters/test_go.py
index 1edffb93..ac7377d9 100644
--- a/tests/interpreters/test_go.py
+++ b/tests/interpreters/test_go.py
@@ -361,6 +361,27 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    interpreter = GoInterpreter()
+
+    expected_code = """
+import "math"
+func score(input []float64) float64 {
+    return sigmoid(2.0)
+}
+func sigmoid(x float64) float64 {
+    if (x < 0.0) {
+        z := math.Exp(x)
+        return z / (1.0 + z)
+    }
+    return 1.0 / (1.0 + math.Exp(-x))
+}"""
+
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_haskell.py b/tests/interpreters/test_haskell.py
index aa2b5224..be64952e 100644
--- a/tests/interpreters/test_haskell.py
+++ b/tests/interpreters/test_haskell.py
@@ -382,6 +382,26 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = r"""
+module Model where
+score :: [Double] -> Double
+score input =
+    sigmoid (2.0)
+sigmoid :: Double -> Double
+sigmoid x
+    | x < 0.0 = z / (1.0 + z)
+    | otherwise = 1.0 / (1.0 + exp (-x))
+  where
+    z = exp x
+"""
+
+    interpreter = HaskellInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_java.py b/tests/interpreters/test_java.py
index c2c28ae9..85b33630 100644
--- a/tests/interpreters/test_java.py
+++ b/tests/interpreters/test_java.py
@@ -589,6 +589,28 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    interpreter = JavaInterpreter()
+
+    expected_code = """
+public class Model {
+    public static double score(double[] input) {
+        return sigmoid(2.0);
+    }
+    private static double sigmoid(double x) {
+        if (x < 0.0) {
+            double z = Math.exp(x);
+            return z / (1.0 + z);
+        }
+        return 1.0 / (1.0 + Math.exp(-x));
+    }
+}"""
+
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_javascript.py b/tests/interpreters/test_javascript.py
index a2e6eb6d..87bbf4c7 100644
--- a/tests/interpreters/test_javascript.py
+++ b/tests/interpreters/test_javascript.py
@@ -373,6 +373,27 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    interpreter = JavascriptInterpreter()
+
+    expected_code = """
+function score(input) {
+    return sigmoid(2.0);
+}
+function sigmoid(x) {
+    if (x < 0.0) {
+        let z = Math.exp(x);
+        return z / (1.0 + z);
+    }
+    return 1.0 / (1.0 + Math.exp(-x));
+}
+"""
+
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_php.py b/tests/interpreters/test_php.py
index 0552857b..7d355fa3 100644
--- a/tests/interpreters/test_php.py
+++ b/tests/interpreters/test_php.py
@@ -375,6 +375,27 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+<?php
+function score(array $input) {
+    return sigmoid(2.0);
+}
+function sigmoid($x) {
+    if ($x < 0.0) {
+        $z = exp($x);
+        return $z / (1.0 + $z);
+    }
+    return 1.0 / (1.0 + exp(-$x));
+}
+"""
+
+    interpreter = PhpInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_powershell.py b/tests/interpreters/test_powershell.py
index d9f6703c..3a209701 100644
--- a/tests/interpreters/test_powershell.py
+++ b/tests/interpreters/test_powershell.py
@@ -415,6 +415,26 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+function Score([double[]] $InputVector) {
+    return Sigmoid $(2.0)
+}
+function Sigmoid([double] $x) {
+    if ($x -lt 0.0) {
+        [double] $z = [math]::Exp($x)
+        return $z / (1.0 + $z)
+    }
+    return 1.0 / (1.0 + [math]::Exp(-$x))
+}
+"""
+
+    interpreter = PowershellInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_python.py b/tests/interpreters/test_python.py
index 62716a1e..e1f05a1f 100644
--- a/tests/interpreters/test_python.py
+++ b/tests/interpreters/test_python.py
@@ -467,6 +467,25 @@ def score(input):
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    interpreter = PythonInterpreter()
+
+    expected_code = """
+import math
+def sigmoid(x):
+    if x < 0.0:
+        z = math.exp(x)
+        return z / (1.0 + z)
+    return 1.0 / (1.0 + math.exp(-x))
+def score(input):
+    return sigmoid(2.0)
+    """
+
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_r.py b/tests/interpreters/test_r.py
index 7f622226..8aba74d4 100644
--- a/tests/interpreters/test_r.py
+++ b/tests/interpreters/test_r.py
@@ -482,6 +482,26 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+sigmoid <- function(x) {
+    if (x < 0.0) {
+        z <- exp(x)
+        return(z / (1.0 + z))
+    }
+    return(1.0 / (1.0 + exp(-x)))
+}
+score <- function(input) {
+    return(sigmoid(2.0))
+}
+"""
+
+    interpreter = RInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_ruby.py b/tests/interpreters/test_ruby.py
index ff9dea60..ff2429b7 100644
--- a/tests/interpreters/test_ruby.py
+++ b/tests/interpreters/test_ruby.py
@@ -386,6 +386,26 @@ def softmax(x)
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+def score(input)
+    sigmoid(2.0)
+end
+def sigmoid(x)
+    if x < 0.0
+        z = Math.exp(x)
+        return z / (1.0 + z)
+    end
+    return 1.0 / (1.0 + Math.exp(-x))
+end
+"""
+
+    interpreter = RubyInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/interpreters/test_visual_basic.py b/tests/interpreters/test_visual_basic.py
index 018aa028..e1f399f2 100644
--- a/tests/interpreters/test_visual_basic.py
+++ b/tests/interpreters/test_visual_basic.py
@@ -613,6 +613,30 @@ def test_softmax_expr():
     utils.assert_code_equal(interpreter.interpret(expr), expected_code)
 
 
+def test_sigmoid_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    expected_code = """
+Module Model
+Function Score(ByRef inputVector() As Double) As Double
+    Score = Sigmoid(2.0)
+End Function
+Function Sigmoid(ByVal number As Double) As Double
+    If number < 0.0 Then
+        Dim z As Double
+        z = Math.Exp(number)
+        Sigmoid = z / (1.0 + z)
+        Exit Function
+    End If
+    Sigmoid = 1.0 / (1.0 + Math.Exp(-number))
+End Function
+End Module
+"""
+
+    interpreter = VisualBasicInterpreter()
+    utils.assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
 def test_reused_expr():
     reused_expr = ast.ExpExpr(ast.NumVal(1.0), to_reuse=True)
     expr = ast.BinNumExpr(reused_expr, reused_expr, ast.BinNumOpType.DIV)
diff --git a/tests/test_ast.py b/tests/test_ast.py
index af43d85a..841df8b3 100644
--- a/tests/test_ast.py
+++ b/tests/test_ast.py
@@ -51,6 +51,7 @@ def test_count_exprs_exclude_list():
             ast.ExpExpr(ast.NumVal(2)),
             ast.LogExpr(ast.NumVal(2)),
             ast.Log1pExpr(ast.NumVal(2)),
+            ast.SigmoidExpr(ast.NumVal(2)),
             ast.SqrtExpr(ast.NumVal(2)),
             ast.PowExpr(ast.NumVal(2), ast.NumVal(3)),
             ast.TanhExpr(ast.NumVal(1)),
@@ -69,6 +70,7 @@ def test_count_exprs_exclude_list():
                 ast.NumVal(6),
                 ast.NumVal(7),
                 ast.NumVal(8),
+                ast.NumVal(9),
                 ast.FeatureRef(1)
             ])),
         ast.BinNumOpType.SUB),
@@ -81,7 +83,7 @@ def test_count_exprs_exclude_list():
 
 
 def test_count_all_exprs_types():
-    assert ast.count_exprs(EXPR_WITH_ALL_EXPRS) == 40
+    assert ast.count_exprs(EXPR_WITH_ALL_EXPRS) == 43
 
 
 def test_exprs_equality():
@@ -102,13 +104,14 @@ def test_exprs_str():
 ExpExpr(NumVal(2.0),to_reuse=False),
 LogExpr(NumVal(2.0),to_reuse=False),
 Log1pExpr(NumVal(2.0),to_reuse=False),
+SigmoidExpr(NumVal(2.0),to_reuse=False),
 SqrtExpr(NumVal(2.0),to_reuse=False),
 PowExpr(NumVal(2.0),NumVal(3.0),to_reuse=False),
 TanhExpr(NumVal(1.0),to_reuse=False),
 BinNumExpr(NumVal(0.0),FeatureRef(0),ADD,to_reuse=False)]),
 IdExpr(SoftmaxExpr(
 NumVal(1.0),NumVal(2.0),NumVal(3.0),NumVal(4.0),NumVal(5.0),
-NumVal(6.0),NumVal(7.0),NumVal(8.0),FeatureRef(1),to_reuse=False),to_reuse=False),SUB),
+NumVal(6.0),NumVal(7.0),NumVal(8.0),NumVal(9.0),FeatureRef(1),to_reuse=False),to_reuse=False),SUB),
 IfExpr(CompExpr(NumVal(2.0),NumVal(0.0),GT),NumVal(3.0),NumVal(4.0)),MUL)
 """.strip().replace("\n", "")
 
diff --git a/tests/test_fallback_expressions.py b/tests/test_fallback_expressions.py
index 261ecf88..1dfee333 100644
--- a/tests/test_fallback_expressions.py
+++ b/tests/test_fallback_expressions.py
@@ -171,3 +171,24 @@ def score(input):
 """
 
     assert_code_equal(interpreter.interpret(expr), expected_code)
+
+
+def test_sigmoid_fallback_expr():
+    expr = ast.SigmoidExpr(ast.NumVal(2.0))
+
+    class InterpreterWithoutSigmoid(PythonInterpreter):
+        sigmoid_function_name = NotImplemented
+
+        def interpret_sigmoid_expr(self, expr, **kwargs):
+            return super(PythonInterpreter, self).interpret_sigmoid_expr(
+                expr, **kwargs)
+
+    interpreter = InterpreterWithoutSigmoid()
+
+    expected_code = """
+import math
+def score(input):
+    return (1.0) / ((1.0) + (math.exp((0.0) - (2.0))))
+"""
+
+    assert_code_equal(interpreter.interpret(expr), expected_code)