Repository: microsoft/knossos-ksc

Issue:
Zero vectors check is slow
The `vec` class used in ksc-generated code contains an `is_zero_` flag which optimizes the case of all-zero vectors. But this adds overhead in cases where all-zero never happens, e.g. GMM objective function is approximately 50% slower when compared to not having the flag (tested with GCC 7). Is there some way to get the best of both worlds here?


Discussion:
When the same vector is accessed multiple times, ideally the C++ compiler would optimize this to only check the flag once. But it doesn't look like this happens with GCC 7 - so most of the slowdown appears to come from repeated checks in `vec<double>::operator[]` when doing basic linear algebra operations.

So an improvement is to provide a way to access the vector elements which bypasses the flag and assumes that the vector is non-zero, and implement the basic linear algebra operations using that (checking `is_zero` once at the start). Though this is still measurably slower than removing the flag completely: it's slower even if you _never_ check the flag.

I'll also run this with GCC10 to see if that does better here.
One possibility: ensure that the `data_` pointer always points to a valid array of zeros, at least in the case of `vec<double>`. This would avoid having to check the flag in `vec<double>::operator[]`. It would be fairly simple/cheap to have every such vector share the same data, to avoid needing to construct an array of zeros each time.
Comparison of the current code with a version with no zero flag (when a vector of zeros is required, this allocates an array of zeros).

<img width="864" alt="objective" src="https://user-images.githubusercontent.com/38362796/86578297-81ea9480-bf73-11ea-8bd7-7dd47f2d3827.png">
<img width="864" alt="jacobian" src="https://user-images.githubusercontent.com/38362796/86578302-83b45800-bf73-11ea-879e-aae44aacbaf4.png">

Similar comparison based on alternative version of `gmm.ks` (moving the call to `gmm_knossos_makeQ` out of the inner loop):

<img width="864" alt="objective" src="https://user-images.githubusercontent.com/38362796/86576109-2d91e580-bf70-11ea-8d91-b28a7340578e.png">
<img width="864" alt="jacobian" src="https://user-images.githubusercontent.com/38362796/86576121-308cd600-bf70-11ea-912a-f275fdba6b3f.png">



Should we close this?
#322 resolves a few easy cases but the `is_zero` flag is still there and does affect performance. From what we've discussed before, I think the plan is to remove the flag. Should we do this now (and then can definitely close this issue)?

At the moment the `is_zero` flag is an overall win on GMM: this is due to the implementation explicitly forming the triangular matrix in `makeQ`, which results in large numbers of zero derivatives. But it's not clear that this one example is a good enough reason to keep the flag in general.

Resolving patch:
diff --git a/src/python/ksc/utils.py b/src/python/ksc/utils.py
--- a/src/python/ksc/utils.py
+++ b/src/python/ksc/utils.py
@@ -148,7 +148,6 @@ def generate_and_compile_cpp_from_ks(ks_str, name_to_call, arg_types, pybind11_p
   py::class_<Class>(m, pyclass_name.c_str(), py::module_local())
     .def(py::init<>())
     .def(py::init([](std::vector<T> const& v) {{ return ks::vec<T>(&g_alloc, v); }}))
-    .def("is_zero",     &Class::is_zero)
     .def("__getitem__", [](const ks::vec<T> &a, const int &b) {{
 	return a[b];
       }})


Test patch:
diff --git a/test/ksc/adbench-lstmpy.cpp b/test/ksc/adbench-lstmpy.cpp
--- a/test/ksc/adbench-lstmpy.cpp
+++ b/test/ksc/adbench-lstmpy.cpp
@@ -23,7 +23,6 @@ void declare_vec(py::module &m, std::string typestr) {
   py::class_<Class>(m, pyclass_name.c_str())
     .def(py::init<>())
     .def(py::init([](std::vector<T> const& v) { return ks::vec<T>(&g_alloc, v); }))
-    .def("is_zero",     &Class::is_zero)
     .def("__getitem__", [](const ks::vec<T> &a, const int &b) {
 	return a[b];
       })
diff --git a/test/ksc/gmmpy.cpp b/test/ksc/gmmpy.cpp
--- a/test/ksc/gmmpy.cpp
+++ b/test/ksc/gmmpy.cpp
@@ -21,7 +21,6 @@ void declare_vec(py::module &m, std::string typestr) {
   py::class_<Class>(m, pyclass_name.c_str())
     .def(py::init<>())
     .def(py::init([](std::vector<T> const& v) { return ks::vec<T>(&g_alloc, v); }))
-    .def("is_zero",     &Class::is_zero)
     .def("__getitem__", [](const ks::vec<T> &a, const int &b) {
 	return a[b];
       })
diff --git a/test/ksc/mnistcnnpy.cpp b/test/ksc/mnistcnnpy.cpp
--- a/test/ksc/mnistcnnpy.cpp
+++ b/test/ksc/mnistcnnpy.cpp
@@ -17,7 +17,6 @@ void declare_vec(py::module &m, std::string typestr) {
   py::class_<Class>(m, pyclass_name.c_str())
     .def(py::init<>())
     .def(py::init([](std::vector<T> const& v) { return ks::vec<T>(&g_alloc, v); }))
-    .def("is_zero",     &Class::is_zero)
     .def("__getitem__", [](const ks::vec<T> &a, const int &b) {
 	return a[b];
       })