DOI: 10.5281/zenodo.3734037
title: Cyclist Actions: Optical Flow Sequences and Trajectories

description: The dataset consists of over 1.1 million samples of labeled cyclists actions. Every sample consists of two optical flow sequences, recorded over the past second (9 optical flow images each), from two different cameras, the past trajectory of the cyclist of the last second (50 past positions), and a label of the currently performed action. The samples were extracted from 1,639 video sequences of cyclists moving across an urban intersection at the University of Applied sciences in Aschaffenburg: https://www.th-ab.de/ueber-uns/organisation/labor/kooperative-automatisierte-verkehrssysteme/ausstattung/ The uploaded files consist of an archive containing 27 numpy files, a single numpy file containing trajectories only, and a json file containing 5-fold cross validation/test split. The numpy files consist of python dictionaries with scenes of the form: <pre><code class="language-python">{SCENE_NAME: 'of_hk1/2': [...], # zip compressed, python pickled optical flow sequences of cameras 1/2 'x/y/z_tracked': [...], # tracked cyclists positions in x/y/z directions, 'x/y/z_smoothed': [...], # smoothed (by rts smoother) cyclists positions in x/y/z directions, 'orientation': [...], # orientation of the cyclists estimated by kalman filters 'ts': [...], # utc timestamps in micro seconds LABEL_NAME: [...], # labels of different actions (0 or 1)}</code></pre> The manually created labels are: straight: cyclists is moving and not turning tr/tl: cyclist is turning left/right move: cyclist is moving with nearly constant velocity and not turning start: cyclist was standing and starts moving starting_movement: first movement of cyclist before starting stop: cyclist was moving/starting and slows down to a halt wait: cyclist is standing hand_signal_left/right: cyclist indicates a turn by hand signal shoulder_check_left/right: cyclist looks over left/right shoulder out_of_saddle: cyclist is standing The optical flow sequences were created using PWC-Net [1]. To extract the zipped/pickled optical flow sequences: <pre><code class="language-python">import cv2 as cv import zlib import pickle import numpy as np # visualize flow def vis_of(of): hsv = np.zeros([of.shape[0], of.shape[1], 3], dtype=np.uint8) hsv[..., 1] = 255 mag, ang = cv.cartToPolar(of[..., 0].astype(np.float32), of[..., 1].astype(np.float32)) hsv[..., 0] = ang * 180 / np.pi / 2 hsv[..., 2] = cv.normalize(mag, None, 0, 255, cv.NORM_MINMAX) bgr = cv.cvtColor(hsv, cv.COLOR_HSV2BGR) return bgr # load npy file from dataset npy_path = 'of_dataset_0.npy' data = np.load(npy_path, allow_pickle=True).item() scene = data[list(data.keys())[0]] # extract optical flow sequence ofs = pickle.loads(zlib.decompress(scene['of_hk1'][i])).astype(np.float16) * 2.0 / 255.0 - 1.0 # show of images in sequence for j in range(len(ofs)): # create bgr image from 2 channel optical flow bgr = vis_of(ofs[j]) cv.imshow("of", bgr) </code></pre> Python code and a description to read the dataset can be found in our GitHub: https://github.com/CooperativeAutomatedTrafficSystemsLab/CyclistActionRecognition [1] D. Sun, X. Yang, M. Liu, and J. Kautz, “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, June 2018, pp. 8934–8943. This work results from the project DeCoInt 2, supported by the German Research Foundation (DFG) within the priority program SPP 1835: "Kooperativ interagierende Automobile", grant numbers DO 1186/1-2, FU 1005/1-2, and SI 674/11-2. Additionally, the work is supported by "Zentrum Digitalisierung Bayern". Due to privacy laws in germany, we are not permitted to publish image sequences.