Repository: autorope/donkeycar

Issue:
Figure out root cause of spikes in control lag time
This was the main issue I experienced at the Feb 18th track day that prevented me from reliably driving the car around the track for training. It always manifests as an intermittent problem but I am also able to observe it at home, although less frequently than I saw it at the track day.

I'm running a local donkey server over wifi, so 4G latency is not a factor here. On wifi, I'm frequently seeing lag times spike above 1s, sometimes as long as 30 or more seconds. The clues that I've seen so far are:

1) This seems to happen more frequently in areas of high network congestion (like the track day when everyone was running nearby).
2) It happens on my home wifi network at least once per minute while driving the donkey car, to varying levels of severity.
3) I tried an alternate router at home on a non-standard wifi channel, and was not able to reproduce the delays.

Here's a sample console log from the pi that shows the spikes. Lag time of ~0.06 is about normal on my home network.

```
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.06897997856140137
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.07510542869567871
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.06453394889831543
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.759141206741333
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.05977487564086914
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.0692141056060791
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.06003284454345703
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.16736602783203125
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.06820440292358398
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.0678567886352539
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.12179088592529297
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.05697226524353027
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.0699162483215332
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.06665158271789551
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.17603182792663574
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 1.0047976970672607
throttle update: 0.0
pulse: 370
angle: 0.0   throttle: 0.0
{"angle": "0", "throttle": "0"}
vehicle <> server: request lag: 0.0619354248046875
throttle update: 0.0
pulse: 370
```


Discussion:
Thanks @alanwells. I remember that the trace shows this hangs on the connection creation. Maybe reusing the connection would prevent these large lags. The requests module provides this functionality with sessions.  http://docs.python-requests.org/en/master/user/advanced/

Resolving patch:
diff --git a/donkey/remotes.py b/donkey/remotes.py
--- a/donkey/remotes.py
+++ b/donkey/remotes.py
@@ -4,6 +4,7 @@
 """
 
 import time
+from datetime import datetime
 import json
 import io
 import os
@@ -32,6 +33,16 @@ def __init__(self, remote_url, vehicle_id='mycar'):
 
         self.control_url = remote_url + '/api/vehicles/control/' + vehicle_id + '/'
         self.last_milliseconds = 0
+        self.session = requests.Session()
+
+        self.log('time,lag\n', write_method='w')
+
+    def log(self, line, path='lag_log.csv', write_method='a'):
+        with open('lag_log.csv', write_method) as f:
+            f.write(line)
+
+
+
         
         
     def decide(self, img_arr, angle, throttle, milliseconds):
@@ -49,31 +60,38 @@ def decide(self, img_arr, angle, throttle, milliseconds):
 
 
         r = None
-
         while r == None:
             #Try connecting to server until connection is made.
+            start = time.time()
             
             try:
-                start = time.time()
-                r = requests.post(self.control_url, 
+                r = self.session.post(self.control_url, 
                                 files={'img': dk.utils.arr_to_binary(img_arr), 
-                                       'json': json.dumps(data)}) #hack to put json in file 
-                end = time.time()
-                lag = end-start
+                                       'json': json.dumps(data)},
+                                       timeout=0.2) #hack to put json in file 
+                
             except (requests.ConnectionError) as err:
                 print("Vehicle could not connect to server. Make sure you've " + 
                     "started your server and you're referencing the right port.")
                 time.sleep(3)
+            
+            except (requests.exceptions.ReadTimeout) as err:
+                print("Request took too long. Retrying")
+                return angle, throttle * .8
+                
+
+        end = time.time()
+        lag = end-start
+        self.log('{}, {} \n'.format(datetime.now().time() , lag ))
+        print('vehicle <> server: request lag: %s' %lag)
 
-        print(r.text)
-        
         data = json.loads(r.text)
         
         angle = float(data['angle'])
         throttle = float(data['throttle'])
         
-        print('vehicle <> server: request lag: %s' %lag)
-
+        
+        
 
         return angle, throttle
 


Test patch: