Repository: StackStorm/st2

Issue:
Sensors have stopped working after upgrade to st2 2.7.1
Hi Stackstorm Dev Team,

Sensors have stopped working in st2 2.7.1 after upgrade from st2 2.6.0

It was giving the error : module "object" has no attribute "poll" whenever the sensor start to initialize. This error was seen in st2sensorcontainer.log

Even the default linux.FileWatchSensor threw errors during sensor initialization when we bring the st2sensorcontainer service

Kindly please look into this issue

Regards
Harish


Discussion:
I can confirm this issue, below are the contents from `/var/log/st2/st2sensorcontainer.log`

```shell
2018-04-23 10:12:09,337 139783376605584 AUDIT access [-] Access granted to "sensors_container" with the token set to expire at "2018-04-24T14:12:09.331961Z". (username='sensors_container',token_expiration='2018-04-24T14:12:09.331961Z')
2018-04-23 10:12:09,339 139783376605584 WARNING api [-] "auth.api_url" configuration option is not configured
2018-04-23 10:12:10,850 140198901300272 INFO sensor_wrapper [-] No config found for sensor "FileWatchSensor"
2018-04-23 10:12:10,852 140198901300272 INFO sensor_wrapper [-] Watcher started
2018-04-23 10:12:10,852 140198901300272 INFO sensor_wrapper [-] Running sensor initialization code
2018-04-23 10:12:10,866 140198901300272 ERROR (unknown file) [-] Traceback (most recent call last):

2018-04-23 10:12:10,868 140198901300272 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py", line 380, in <module>

2018-04-23 10:12:10,868 140198901300272 ERROR (unknown file) [-]     
2018-04-23 10:12:10,869 140198901300272 ERROR (unknown file) [-] obj.run()

2018-04-23 10:12:10,870 140198901300272 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2reactor/container/sensor_wrapper.py", line 237, in run

2018-04-23 10:12:10,870 140198901300272 ERROR (unknown file) [-]     
2018-04-23 10:12:10,871 140198901300272 ERROR (unknown file) [-] self._sensor_instance.setup()

2018-04-23 10:12:10,872 140198901300272 ERROR (unknown file) [-]   File "/opt/stackstorm/packs/linux/sensors/file_watch_sensor.py", line 17, in setup

2018-04-23 10:12:10,873 140198901300272 ERROR (unknown file) [-]     
2018-04-23 10:12:10,874 140198901300272 ERROR (unknown file) [-] self._tail = Tail(filenames=[])

2018-04-23 10:12:10,874 140198901300272 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/logshipper/tail.py", line 74, in __init__

2018-04-23 10:12:10,875 140198901300272 ERROR (unknown file) [-]     
2018-04-23 10:12:10,875 140198901300272 ERROR (unknown file) [-] self.watch_manager)

2018-04-23 10:12:10,876 140198901300272 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/logshipper/pyinotify_eventlet_notifier.py", line 34, in __init__

2018-04-23 10:12:10,876 140198901300272 ERROR (unknown file) [-]     
2018-04-23 10:12:10,877 140198901300272 ERROR (unknown file) [-] read_freq=0, threshold=0, timeout=None)

2018-04-23 10:12:10,877 140198901300272 ERROR (unknown file) [-]   File "/opt/stackstorm/st2/lib/python2.7/site-packages/pyinotify.py", line 1138, in __init__

2018-04-23 10:12:10,878 140198901300272 ERROR (unknown file) [-]     
2018-04-23 10:12:10,879 140198901300272 ERROR (unknown file) [-] self._pollobj = select.poll()

2018-04-23 10:12:10,879 140198901300272 ERROR (unknown file) [-] AttributeError
2018-04-23 10:12:10,880 140198901300272 ERROR (unknown file) [-] : 
2018-04-23 10:12:10,881 140198901300272 ERROR (unknown file) [-] 'module' object has no attribute 'poll'
2018-04-23 10:12:10,882 140198901300272 ERROR (unknown file) [-] 

2018-04-23 10:12:10,883 140198901300272 INFO sensor_wrapper [-] Stopping trigger watcher
2018-04-23 10:12:10,889 140198901300272 INFO sensor_wrapper [-] Invoking cleanup on sensor
2018-04-23 10:12:14,323 139783376605264 INFO process_container [-] Process for sensor linux.FileWatchSensor has exited with code 1
```
Something else is going on with that file watch sensor - I suspect something related to pyinotify. I think that's a red herring here. 

Testing with a passive sensor - e.g. github.com/StackStorm/githubwebhooksensor, it is running OK. 

@harishgopalan which sensor are you having problems with? Is this your own custom sensor? Are you using pyinotify? 

There *could* be something going on related to `select.poll()`, but this only applies to those sensors that use this. 
Hi Lindsay,

We've a standalone sensor which monitors some failures and dispatches
trigger and then wait for X minute before checking again.

After upgrade to 2.7.1 we noticed that the sensors are not running, when we
looked at the st2sensorcontainer.log, we noticed this error is occurring
across all sensors

Error : module 'object' has no attribute 'poll'

Regards
Harish

On Thu, Apr 26, 2018 at 3:45 AM, Lindsay Hill <notifications@github.com>
wrote:

> Something else is going on with that file watch sensor - I suspect
> something related to pyinotify. I think that's a red herring here.
>
> Testing with a passive sensor - e.g. github.com/StackStorm/
> githubwebhooksensor, it is running OK.
>
> @harishgopalan <https://github.com/harishgopalan> which sensor are you
> having problems with?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/StackStorm/st2/issues/4094#issuecomment-384451073>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACHQNguBFr_H439CwawCp6V2EmJH_TfNks5tsPVogaJpZM4Tf_Y6>
> .
>

Can you provide a specific example other than the filewatch sensor? I need to see the actual code
@harishgopalan can you please provide a code example here? 

Resolving patch:
diff --git a/st2common/st2common/util/monkey_patch.py b/st2common/st2common/util/monkey_patch.py
--- a/st2common/st2common/util/monkey_patch.py
+++ b/st2common/st2common/util/monkey_patch.py
@@ -45,19 +45,42 @@ def monkey_patch():
     eventlet.monkey_patch(os=True, select=True, socket=True, thread=patch_thread, time=True)
 
 
-def use_select_poll_workaround():
+def use_select_poll_workaround(nose_only=True):
     """
-    Work around for some tests which injects original select module with select.poll()
-    available to sys.modules.
+    Work around for eventlet monkey patched code which relies on select.poll() functionality
+    which is blocking and not green / async.
+
+    Keep in mind that this was always the case in old eventlet versions, but v0.20.0 removed
+    select.poll() to avoid users shooting themselves in the foot and rely on blocking code.
+
+    This code inserts original blocking select.poll() into the sys.modules so it can be used
+    where needed.
+
+    Keep in mind that this code should only be uses in scenarios where blocking is not a problem.
+
+    For example:
+
+    1) Inside tests which rely on subprocess.poll() / select.poll().
+    2) Inside code which relies on 3rd party libraries which use select.poll() and are ran
+       inside a subprocess so blocking is not an issue (e.g. actions, sensors).
+
+    :param nose_only: True to only perform monkey patch when running tests under nose tests
+                      runner.
+    :type nose_only: ``bool``
     """
     import sys
+    import select
     import subprocess
     import eventlet
 
     # Work around to get tests to pass with eventlet >= 0.20.0
-    if 'nose' in sys.modules.keys():
-        sys.modules['select'] = eventlet.patcher.original('select')
-        subprocess.select = eventlet.patcher.original('select')
+    if not nose_only or (nose_only and'nose' in sys.modules.keys()):
+        # Add back blocking poll() to eventlet monkeypatched select
+        original_poll = eventlet.patcher.original('select').poll
+        select.poll = original_poll
+
+        sys.modules['select'] = select
+        subprocess.select = select
 
         if sys.version_info >= (3, 6, 5):
             # If we also don't patch selectors.select, it will fail with Python >= 3.6.5
diff --git a/st2reactor/st2reactor/container/sensor_wrapper.py b/st2reactor/st2reactor/container/sensor_wrapper.py
--- a/st2reactor/st2reactor/container/sensor_wrapper.py
+++ b/st2reactor/st2reactor/container/sensor_wrapper.py
@@ -37,8 +37,9 @@
 from st2reactor.sensor.base import PollingSensor
 from st2reactor.sensor import config
 from st2common.services.datastore import SensorDatastoreService
-from st2common.util.monkey_patch import monkey_patch
 from st2common.validators.api.reactor import validate_trigger_payload
+from st2common.util.monkey_patch import monkey_patch
+from st2common.util.monkey_patch import use_select_poll_workaround
 
 __all__ = [
     'SensorWrapper',
@@ -46,6 +47,7 @@
 ]
 
 monkey_patch()
+use_select_poll_workaround(nose_only=False)
 
 
 class SensorService(object):


Test patch:
diff --git a/st2reactor/tests/unit/test_sensor_wrapper.py b/st2reactor/tests/unit/test_sensor_wrapper.py
--- a/st2reactor/tests/unit/test_sensor_wrapper.py
+++ b/st2reactor/tests/unit/test_sensor_wrapper.py
@@ -3,6 +3,10 @@
 import unittest2
 
 import mock
+import eventlet
+
+from st2common.util.monkey_patch import monkey_patch
+monkey_patch()
 
 import st2tests.config as tests_config
 from st2tests.base import TESTS_CONFIG_PATH
@@ -136,3 +140,14 @@ def test_sensor_init_fails_sensor_code_contains_typo(self):
             self.assertTrue('line 5, in <module>' in str(e))
         else:
             self.fail('NameError not thrown')
+
+    def test_sensor_wrapper_poll_method_still_works(self):
+        # Verify that sensor wrapper correctly applied select.poll() eventlet workaround so code
+        # which relies on select.poll() such as subprocess.poll() still works
+        # Note: If workaround is not applied "AttributeError: 'module' object has no attribute
+        # 'poll'" will be thrown
+        import select
+
+        self.assertTrue(eventlet.patcher.is_monkey_patched(select))
+        self.assertTrue(select != eventlet.patcher.original('select'))
+        self.assertTrue(select.poll())