Package: transbigdata
English中文版TransBigDataIntroductionTransBigDatais a Python package developed for transportation spatio-temporal big data processing, analysis and visualization.TransBigDataprovides fast and concise methods for processing common transportation spatio-temporal big data such as Taxi GPS data, bicycle sharing data and bus GPS data.TransBigDataprovides a variety of processing methods for each stage of transportation spatio-temporal big data analysis. The code withTransBigDatais clean, efficient, flexible, and easy to use, allowing complex data tasks to be achieved with concise code.For some specific types of data,TransBigDataalso provides targeted tools for specific needs, such as extraction of Origin and Destination(OD) of taxi trips from taxi GPS data and identification of arrival and departure information from bus GPS data. The latest stable release of the software can be installed via pip and full documentation
can be found athttps://transbigdata.readthedocs.io/en/latest/. Introduction PPT can be foundhereandhere(in Chinese)Target AudienceThe target audience ofTransBigDataincludes:Data science researchers and data engineers in the field of transportation big data, smart transportation systems, and urban computing, particularly those who want to integrate innovative algorithms into intelligent trasnportation systemsGovernment, enterprises, or other entities who expect efficient and reliable management decision support through transportation spatio-temporal data analysis.Technical FeaturesProvide a variety of processing methods for each stage of transportation spatio-temporal big data analysis.The code withTransBigDatais clean, efficient, flexible, and easy to use, allowing complex data tasks to be achieved with concise code.Main FunctionsCurrently,TransBigDatamainly provides the following methods:Data Quality: Provides methods to quickly obtain the general information of the dataset, including the data amount the time period and the sampling interval.Data Preprocess: Provides methods to clean multiple types of data error.Data Gridding: Provides methods to generate multiple types of geographic grids (Rectangular grids, Hexagonal grids) in the research area. Provides fast algorithms to map GPS data to the generated grids.Data Aggregating: Provides methods to aggregate GPS data and OD data into geographic polygon.Data Visualization: Built-in visualization capabilities leverage the visualization package keplergl to interactively visualize data on Jupyter notebook with simple code.Trajectory Processing: Provides methods to process trajectory data, including generating trajectory linestring from GPS points, and trajectory densification, etc.Basemap Loading: Provides methods to display Mapbox basemap on matplotlib figuresGrid processing framework offered by TransBigDataHere is an overview of the gridding framework offered byTransBigData.SeeThis Examplefor further details.Trajectory processing framework offered by TransBigDataHere is an overview of the Trajectory processing framework offered byTransBigData.SeeThis Examplefor further details.InstallationTransBigDatasupport Python >= 3.6Using pypiTransBigDatacan be installed by usingpip install. Before installingTransBigData, make sure that you have installed the availablegeopandas package. If you already have geopandas installed, run the following code directly from the command prompt to installTransBigData:pip install transbigdataUsing conda-forgeYou can also installTransBigDatabyconda-forge, this will automaticaly solve the dependency, it can be installed with:conda install -c conda-forge transbigdataContributing to TransBigDataAll contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome. A detailed overview on how to contribute can be found in thecontributing guideon GitHub.ExamplesExample of data visualizationVisualize trajectories (with keplergl)Visualize data distribution (with keplergl)Visualize OD (with keplergl)Example of taxi GPS data processingThe following example shows how to use theTransBigDatato perform data gridding, data aggregating and data visualization for taxi GPS data.Read the dataimporttransbigdataastbdimportpandasaspd#Read taxi gps datadata=pd.read_csv('TaxiData-Sample.csv',header=None)data.columns=['VehicleNum','time','lon','lat','OpenStatus','Speed']dataVehicleNumtimelonlatOpenStatusSpeed03474520:27:43113.80684722.62324912713474520:24:07113.80989822.6273990023474520:24:27113.80989822.6273990033474520:22:07113.81134822.6280670043474520:10:06113.81988522.647800054.....................5449942826521:35:13114.32150322.7094990185449952826509:08:02114.32270122.681700005449962826509:14:31114.33670022.690100005449972826521:19:12114.35260022.728399005449982826519:08:06114.13770322.62170000544999 rows × 6 columnsData pre-processingDefine the study area and use thetbd.clean_outofboundsmethod to delete the data out of the study area#Define the study areabounds=[113.75,22.4,114.62,22.86]#Delete the data out of the study areadata=tbd.clean_outofbounds(data,bounds=bounds,col=['lon','lat'])Data griddingThe most basic way to express the data distribution is in the form of geograpic grids.TransBigDataprovides methods to generate multiple types of geographic grids (Rectangular grids, Hexagonal grids) in the research area. For rectangular gridding, you need to determine the gridding parameters at first (which can be interpreted as defining a grid coordinate system):#Obtain the gridding parametersparams=tbd.area_to_params(bounds,accuracy=1000)params{'slon': 113.75,
'slat': 22.4,
'deltalon': 0.00974336289289822,
'deltalat': 0.008993210412845813,
'theta': 0,
'method': 'rect',
'gridsize': 1000}The gridding parameters store the information of the initial position, the size and the angle of the gridding system.The next step is to map the GPS data to their corresponding grids. Using thetbd.GPS_to_grid, it will generate theLONCOLcolumn and theLATCOLcolumn (Rectangular grids). The two columns together can specify a grid:#Map the GPS data to gridsdata['LONCOL'],data['LATCOL']=tbd.GPS_to_grid(data['lon'],data['lat'],params)Count the amount of data in each grids, generate the geometry of the grids and transform it into a GeoDataFrame:#Aggregate data into gridsgrid_agg=data.groupby(['LONCOL','LATCOL'])['VehicleNum'].count().reset_index()#Generate grid geometrygrid_agg['geometry']=tbd.grid_to_polygon([grid_agg['LONCOL'],grid_agg['LATCOL']],params)#Change the type into GeoDataFrameimportgeopandasasgpdgrid_agg=gpd.GeoDataFrame(grid_agg)#Plot the gridsgrid_agg.plot(column='VehicleNum',cmap='autumn_r')Triangle and Hexagon grids & rotation angleTransBigDataalso support the triangle and hexagon grids. It also supports given rotation angle for the grids. We can alter the gridding parameter:#set to the hexagon gridsparams['method']='hexa'#or set as triangle grids: params['method'] = 'tri'#set a rotation angle (degree)params['theta']=5Then we can do the GPS data matching again:#Triangle and Hexagon grids requires three columns to store IDdata['loncol_1'],data['loncol_2'],data['loncol_3']=tbd.GPS_to_grid(data['lon'],data['lat'],params)#Aggregate data into gridsgrid_agg=data.groupby(['loncol_1','loncol_2','loncol_3'])['VehicleNum'].count().reset_index()#Generate grid geometrygrid_agg['geometry']=tbd.grid_to_polygon([grid_agg['loncol_1'],grid_agg['loncol_2'],grid_agg['loncol_3']],params)#Change the type into GeoDataFrameimportgeopandasasgpdgrid_agg=gpd.GeoDataFrame(grid_agg)#Plot the gridsgrid_agg.plot(column='VehicleNum',cmap='autumn_r')Data Visualization(with basemap)For a geographical data visualization figure, we still have to add the basemap, the colorbar, the compass and the scale. Usetbd.plot_mapto load the basemap andtbd.plotscaleto add compass and scale in matplotlib figure:importmatplotlib.pyplotaspltfig=plt.figure(1,(8,8),dpi=300)ax=plt.subplot(111)plt.sca(ax)#Load basemaptbd.plot_map(plt,bounds,zoom=11,style=4)#Define colorbarcax=plt.axes([0.05,0.33,0.02,0.3])plt.title('Data count')plt.sca(ax)#Plot the datagrid_agg.plot(column='VehicleNum',cmap='autumn_r',ax=ax,cax=cax,legend=True)#Add scaletbd.plotscale(ax,bounds=bounds,textsize=10,compasssize=1,accuracy=2000,rect=[0.06,0.03],zorder=10)plt.axis('off')plt.xlim(bounds[0],bounds[2])plt.ylim(bounds[1],bounds[3])plt.show()Citation informationPlease citethiswhen usingTransBigDatain your research. Citation information is as follows:@article{Yu2022,doi={10.21105/joss.04021},url={https://doi.org/10.21105/joss.04021},year={2022},publisher={The Open Journal},volume={7},number={71},pages={4021},author={Qing Yu and Jian Yuan},title={TransBigData: A Python package for transportation spatio-temporal big data processing, analysis and visualization},journal={Journal of Open Source Software}}Introducing Video (In Chinese)BilibiliYoutube