Package: itsim-project-creation-library
Itsim project creation libraryHow to use[itsim-project-data] $ pip install itsim_project_creation_libraryThen in python:fromitsim_project_creation_libraryimport(feed_processing,gis_processing,log_progress,)You can also import aliases:fromitsim_project_creation_libraryimport(feedp,gisp,log_progress,)log_progressis just something to print progress on stdout. Same syntax asprint.How to develop on itpip install -e .How to create a new ItSim projectThis guide covers the steps necessary to create a new project in the ItSim application. This document is intended for developers and people with skills in Python and data processing. Most of the processing operations are done thanks to the Pandas and GeoPandas libraries.Prerequisites for a new ItSim projectFirst of all, to create a new project, we will need:Feed input data: Zero, one or several GTFS file(s). In these files we find the description of the transport plan and network.Geographic Information System (GIS) input data: Zero, one or several shapefiles (.shp). These files will contain geographical data.A filled-outproject creation formthat contains parameters for the project.Processing feed input dataIn this section, we will see how to check feed input data and how to correct the file(s) if required.Check validity of a GTFSFirst, it is necessary to check the validity of the given GTFS with Google'sFeed validator(for Windows or Linux). Once downloaded, you can execute it and drag & drop your GTFS file into it. The software will check the compatibility of the file with theGTFS Reference.After a while, you will get the validity report displayed in your web browser. Then, you will have to check the report and its error and warning messages. Note that errors will most likely cause problems when loading the GTFS into ItSim. That is the reason why the GTFS must be corrected with a processing script and using some of the processing functions of the ItSim project creation library.Warnings are less likely to cause problems but it is recommended to check each message to ensure that it will not be harmful to the import.Check the GTFS contentAfter checking validity of the GTFS file, we would like to have a better idea of its content. We might ask ourselves many questions:How much routes does the network have?Which transport mode(s) does the network have?Do the routes have geometries (shapes), and are they accurate?To visualize the GTFS in software like QGIS, it should be converted to a GeoJSON file. You can convert your GTFS file with thegtfs_to_geojsonscript, using eitherpipenv shellorpipenv run.Example:[itsim-project-data] $ gtfs_to_geojson -g path/to/mycity/MyCity_GTFS_2020.zip -j path/to/mycity/test.jsonOnce it is done, you can import the generated GeoJSON file into QGIS and check its content.Write a feed processing fileTo correct the GTFS file, create agtfs_generator.pyfile in a directory that has the name of your project in theitsm-project-data's repository.Here is an example of how the file should be structured. In this case we are removing shapes of routes and group routes:fromitsim_project_creation_libraryimportfeedp,log_progressfrompathlibimportPathfromosimportenvironfromgtfstkimportread_gtfs,write_gtfsif__name__=="__main__":# Initialize paths# path of the local project directoryPROJECT_DATA_PATH=Path(__file__).resolve().parent# path of the volume in which the original data is savedITSIM_DATA_PATH=Path(environ.get('ITSIM_DATA_PATH','/mnt/systra/G'))# path to the original data foldermycity_path=(ITSIM_DATA_PATH/"PLT/ZZ_DCME/3_GESTION/7_LE LAB DIGITAL/800_PROJETS/PROJET_20160622_ITSIM/1_TRAVAIL/6_RUN/1_Données d'entrée/ItSim/MyCity")# path to the GTFS to usemycity_gtfs_path=mycity_path/'GTFS'/'MyCity_GTFS_2020.zip'# output pathgtfs_output_path=PROJECT_DATA_PATH/'mycity-gtfs.zip'# Define bboxbbox={'north_lat':48.902192,'south_lat':48.814099,'west_lon':2.250824,'east_lon':2.425232,}# Processingnb_of_steps=3log_progress(f"Step 1/{nb_of_steps}: Read GTFS")feed=read_gtfs(mycity_gtfs_path,dist_units='km')log_progress(f"Step 2/{nb_of_steps}: Remove shapes")new_feed=feedp.remove_shapes(feed=feed)log_progress(f"Step 3/{nb_of_steps}: Group routes")new_feed=feedp.group_routes(feed=new_feed)# You can add as much operations as required and even yours…write_gtfs(new_feed,gtfs_output_path)log_progress(f"GTFS file{gtfs_output_path}created!")Operations on the feedIn this section, you will find all the recurring operations you might need if you have to correct your GTFS.Merge several feedsItSim can only load one GTFS when creating a new project, but we might be given several. For instance, train lines and bus lines might be split into two distinct GTFS files, simply because bus and train lines are operated by two different companies.To import data, we will have to merge these files into one. To do so, we will use themerge_gtfs()function. In the following example, GTFS files contained in the directory provided as an argument will be merged in one single GTFS file:gtfs_path="path/to/feed/folder"feed=merge_gtfs(origin_gtfs_path=gtfs_path)If two elements have the same ID, a suffix is added. It consists of an underscore with a number. An element with the ID12will have the ID12_1afterwards if there was already an element with the same ID.Get a feed sample based on a bounding boxSometimes, we might want to use a sample of a large feed. For instance, we might only have at our disposal a GTFS that covers a full region or country. However, we might only want to use data that cover a single city of this region.To get a sample from a large feed, the first thing we will have to do is to define a bounding box. To help you define your bounding box, you canuse this website.Then, useget_feed_sample_from_bbox()function to only keep a sample of the given feed, like in the following example:gtfs_path="path/to/feed/folder/feed.zip"feed=read_gtfs(gtfs_path,dist_units='km')bbox={'north_lat':48.902192,'south_lat':48.814099,'west_lon':2.250824,'east_lon':2.425232,}feed=feedp.get_feed_sample_from_bbox(feed=feed,bbox=bbox)At first, this function will detect the stops that are located in the zone defined by the bounding box. Then, it will keep only the routes that haveall their stopsin the zone.Filter routes based on transport modesSometimes, we would like to work with only one or several specific transport mode(s). For example, we would like to work only on the Bus network, but the original feed also contains Light rail and Rail routes. The functionroute_types_filter()will help you do that.In this example, we would like to remove Light rail (0) and Rail (2) routes:gtfs_path="path/to/feed/folder/feed.zip"feed=read_gtfs(gtfs_path,dist_units='km')route_types=[0,2]new_feed=feedp.route_types_filter(feed=feed,route_types=route_types,filter_type='remove'):Note that thefilter_typecould be'keep'or'remove'.The transport modes are indicated using theroute_typevalues described in theGTFS Reference.Filter routes based on short namesremove_routes_from_feed_by_short_names()feed,routes_to_remove_short_names=['bad route A','bad route B']new_feed=feedp.remove_routes_from_feed_by_short_names(feed,routes_to_remove_short_names)Regroup routesIn our GTFS, some patterns that describe the same route could be dispatched between several ones. This case is not always obvious to detect but the fact that many routes have the same name in your network could be a clue.
You can also check by comparing your network with its map given by the operator. Google is your friend! (or not).To group the routes by name, we can use thegroup_routes()function as follow:gtfs_path="path/to/feed/folder/feed.zip"feed=read_gtfs(gtfs_path,dist_units='km')new_feed=feedp.group_routes(feed=feed)Remove geometries (shapes)Sometimes, shapes that are given in our GTFS might be only made of straight lines or be completely inaccurate. In this case, we most likely want to regenerate these shapes when the project is imported in ItSim. To enable ItSim to generate shapes for lines that will follow the road network (buses…), we will have to remove all the existing shapes.To do so, we will use theremove_shapes()function. In this example, the geometries will be removed from the given feed.gtfs_path="path/to/feed/folder/feed.zip"feed=read_gtfs(gtfs_path,dist_units='km')new_feed=feedp.remove_shapes(feed=feed)Processing GIS dataThere are several GIS data types:surfaces_density: for zonal data expressed in densitiesdots_value: represents dots on the map with an absolute valuelines_and_dots: to use for layers that are not densities or valuesCheck content of GIS dataIt is possible to easily check the content of a shapefile by importing it directly into QGIS. In QGIS, you could check the projection in which the layer is expressed by having a look at its properties.Also take a look at the attribute table to have a better idea of how the data are expressed.Here is a list of some items to check before processing shapefiles:Which type of shapefile (zonal, dots…)?Which columns are useful? Which are not?How are values expressed (absolute, density)?Do values seem coherent? (maximum, minimum, spread)Are there missing values?Which properties are already included in the file? Which have to be set?Operations on GIS dataIn this section, you will find all the recurring operations you might need in case you have to correct your GIS data.Reproject dataItSim only manages layers expressed in theWGS84 projection(EPSG:4326), which is the standard projection for GPS systems. However, many of the layers we have at our disposal might be expressed in the projection used by the agency that created the data. As an example, theLambert93 projection (EPSG:2154) is the projection used for most of the layers in metropolitan France. This is due to the fact that this projection gives accurate areas for this latitude.To reproject data, you can use thereproject_layer_in_WGS84()function which will reproject the data in WGS84 like in our example:gis_data_path="path/to/gis_files/pop.shp"data=read_file(gis_data_path)new_data=gisp.reproject_layer_in_WGS84(data)Note that you should not use the WGS84 projection to calculate areas.It is advised to project in WGS84 at the end of your process.Calculate areaIn case you need to process a zonal layer but the areas of the zones are missing, it is possible to generate them by using thecalculate_area()function.gis_data_path="path/to/gis_files/pop.shp"data=read_file(gis_data_path)new_data=gisp.calculate_area(data)ThedataDataFrame will contain anareacolumn with area expressedin km². Note that if thearea_unit_in_km2option isFalse,areawill be expressed in m².ItSim expects density expressed in/km², so be careful if you use areas expressed inm².The resultDataFramewill be projected inGall-Petersprojection, this corresponds toSR-ORG:22projection.Calculate densitiesData in zonal shapefiles could be expressed in absolute values but ItSim only manages data expressed in densities. In this case, it will be necessary to convert data into densities values by using thecalculate_density()function.gis_data_path="path/to/gis_files/pop.shp"data=read_file(gis_data_path)new_data=gisp.calculate_density(data=data,field_from='pop',field_to='pop_dens',area_field='area',)Thenew_dataDataFrame will contain apop_denscolumn expressed in[unit]/km². Theareacolumn is used for the input area andshould be expressed in km²unlessarea_field_unit_in_km2option isFalse, then it is expected to be in m².If noarea_fieldis indicated, the area is calculated in this function and thearea_field_unit_in_km2parameter is ignored.The resultDataFramewill be projected inGall-Petersprojection, this corresponds toSR-ORG:22projection.Bounding box and coordinatesYou can easily generate a bounding box thanks to thecreate_bbox()function by only giving the coordinates of the zone to define.Example:min_lat=48.814099max_lat=48.902192min_lon=2.250824max_lon=2.425232bbox=gisp.create_bbox(min_lat=min_lat,max_lat=max_lat,min_lon=min_lon,max_lon=max_lon)Otherwise, you can retrieve coordinates from a bounding box with theget_coordinates_from_bbox()function.Example:bbox={'north_lat':48.902192,'south_lat':48.814099,'west_lon':2.250824,'east_lon':2.425232,}(min_lat,max_lat,min_lon,max_lon)=gisp.get_coordinates_from_bbox(bbox=bbox)Merge layersIn case we have several data sources at our disposal, we can merge several shapefiles into one by using themerge_and_clean_layers()function. However, be careful, this function can only be used with layers of the same structure (same type, same columns).gis_data_path="path/to/gis_files"raw_data_paths=[gis_data_path+"/pop_76.shp",gis_data_path+"/pop_27.shp"]columns_to_keep=['pop_dens','superficie']population_data=gisp.merge_and_clean_layers(raw_data_paths,*columns_to_keep)Note that even if we only keep columns that are indicated in thecolumns_to_keeplist, the function will also keep thegeometrycolumn.Generate a layer from carroyageIf you are looking for population data on metropolitan France and you do not have any other sources, you can use the INSEE's "carroyage": a georeferenced grid with population density data. To generate a population shapefile from the carroyage, you can use thecreate_carroyage()function.# Define paths for carroyagecarroyage_root=(ITSIM_DATA_PATH/"PLT/ZZ_DCME/3_GESTION/7_LE LAB DIGITAL/800_PROJETS/PROJET_20160622_ITSIM"/"1_TRAVAIL/2_DEVELOPPEMENT/1.1_Données SYSTRA/France")carroyage_path=carroyage_root/"carroyage_insee_200m_2010"/"200m-carreaux-metropole"/"car_m.mif"population_path=carroyage_root/"carroyage_insee_200m_2010"/"200m-carreaux-metropole"/"car_m.dbf"# Define output pathpopulation_output_path=Path("/path/to/mycity/population.shp")bbox={'north_lat':48.902192,'south_lat':48.814099,'west_lon':2.250824,'east_lon':2.425232,}pop_data=gisp.create_carroyage(carroyage_path=carroyage_path,data_path=population_path,output_path=population_output_path,bbox=bbox,)It is also possible to give coordinates as parameters instead of bbox:# Define paths for carroyagecarroyage_root=(ITSIM_DATA_PATH/"PLT/ZZ_DCME/3_GESTION/7_LE LAB DIGITAL/800_PROJETS/PROJET_20160622_ITSIM"/"1_TRAVAIL/2_DEVELOPPEMENT/1.1_Données SYSTRA/France")carroyage_path=carroyage_root/"carroyage_insee_200m_2010"/"200m-carreaux-metropole"/"car_m.mif"population_path=carroyage_root/"carroyage_insee_200m_2010"/"200m-carreaux-metropole"/"car_m.dbf"# Define output pathpopulation_output_path=Path("/path/to/mycity/population.shp")min_lat=48.814099max_lat=48.902192min_lon=2.250824max_lon=2.425232pop_data=gisp.create_carroyage(carroyage_path=carroyage_path,data_path=population_path,output_path=population_output_path,min_lat=min_lat,max_lat=max_lat,min_lon=min_lon,max_lon=max_lon)Customize GIS layers aspectIt is possible to customize parameters to changes the look ofline_and_dotsGIS layers in ItSim. There are functions for each type of layer (dots, lines, shapes). When setting properties, you can choose to use already existing values that are in one of the layer's columns or a single value that will be used for every entry.DotsDots layers are mostly used to represent interest points, stations, buildings… You can use use theset_dot_properties()function to customize the following parameters:Title (text that will be displayed into a popup)ColorRadiusOpacityIn the following example, I already have the information for names, so I will give the name of the column to use as a data source. Meanwhile, I do not have any information about the colors, so I would like to set a color for every entry. in this case, all I need to do is to create a callable (a lambda function below) that will be applied to every entry. I also would like to set a specific value for the radius and for the opacity, so I will just provide the value to apply to every entry:data=read_file('path/to/my/file')dot_title_field='NAME'dot_color_function=lambdal:'#A15C33'data_with_props=gisp.set_dots_properties(data=data,title=dot_title_field,color=dot_color_function,radius=5,opacity=0.8,):LinesLine layers are mostly used to represent an uneditable transport network. For example, it could be used to represent a railway network that interacts with the bus network we are editing. It could also be usedYou can use theset_lines_properties()function to customize the following parameters:TitleColorRoute type (the route type will influence the lines' width as they are displayed on the map; use it only if it represents a transport network)WidthOpacityIn the following example, I would like to compute the title based on two existing columns in my dataframe. To do so, I will give a lambda that will return a string from the two other existing columns. In the meantime, I would like to set a color for the lines to display but I do not have any information about the colors in my dataframe. This is why I will set a color based on random color for theroute_colorparameter.fromrandomimportrandintdata=read_file('path/to/my/file')line_title_field=lambdal:l['name']+' - '+l['neighbourhood']colors=['#169f5c','#345b64','#543c10','#8eea21','#116f95','#cf26a9','#a1d188','#c0c577','#daef5d','#50c04e','#16ce77']nb_colors=len(colors)line_color_function=lambdal:colors[randint(1,nb_colors)-1]data_with_props=gisp.set_lines_properties(data=data,title=line_title_field,color=line_color_function,opacity=0.8,route_type=3,)Note that if thewidthparameter is provided, it will override theroute_type's computed width.ShapesShapes layers are used to represents zones on the map. They could be used to represent buildings, project areas, facilities…You can use theset_shapes_properties()function to customize the following parameters:TitleFill colorStroke colorOpacityIn the following example, we will retrieve the name from the already existing data, set colors for fill and stroke of the shape with callables (lambda functions) and a single value for opacity.data=read_file('path/to/my/file')shape_title_field='NAME'shape_fill_color_function=lambdax:'#2A7AE0'shape_fill_color_function=lambdax:'#1959A0'data_with_props=gisp.set_shapes_properties(data=data,title=shape_title_field,fill_color=shape_fill_color_function,stroke_color=shape_stroke_color_function,opacity=0.5,)Property fieldsIn case you need to get the names of the properties' fields in order to clean a layer's columns, you can use the following functions to get them:get_dot_properties_names()get_line_properties_names()get_shape_properties_names()Write a project creation scriptTo create a project, ItSim's back-end will need:Aproject description JSONfile(mandatory)Avalid GTFSfile (optional)One or severalshapefile(s)(optional)In order to generate all the required files in a single command, we will write a shell script at "itsim-project-data" repository's root.In this example, we will create a project "MyCity" based on a GTFS and on a single zonal population shapefile. The first thing is to write acreate_mycity.shscript calling data processing scripts in case files are missing:#!/bin/shset-ePROJECTS_DATA_PATH="$(dirname$(readlink-f"$0"))"# Creating GTFS archive if neededif[!-f"$PROJECTS_DATA_PATH/mycity/mycity-gtfs.zip"];thenecho"Generate missing GTFS file..."pipenvrunpython$PROJECTS_DATA_PATH/mycity/gtfs_generator.pyfi# Creating shapefiles if neededNEW_SHP_GENERATED=falseif[!-f"$PROJECTS_DATA_PATH/mycity/mycity_population_2020.shp"];thenecho"Generate missing shapefiles..."python$PROJECTS_DATA_PATH/mycity/data_processing.pyNEW_SHP_GENERATED=truefiThen, zonal layers should be simplified if needed. This step reduces shapefile size and geometry complexity:# Simplifying shapefiles if needed (updated shapefiles or missing simplified versions)if[!-f"$PROJECTS_DATA_PATH/mycity/mycity_population_2020.shp"]||$NEW_SHP_GENERATED;thenecho"Simplifying shapes..."simplify_shapefile"$PROJECTS_DATA_PATH/mycity/mycity_population_2020.shp"pop_dens300.9"$PROJECTS_DATA_PATH/mycity/mycity_population_2020_simplified.shp"fiAfterwards, we will determine the legend of density layers. This is an operation to carry out by hand on simplified layers. It is important to check the accuracy of results given by the./scripts/shp2geojsonscript and round up them if needed:# Generating legends from simplified layers### Color legends scales# Population colors: FEE5D9-A50F15 (red)# shp2geojson -i $PROJECTS_DATA_PATH/mycity/mycity_population_2020_simplified.shp -f pop_dens -c FEE5D9 -C A50F15# => [Enter the results given by the script]# rounded => [Enter the legend to use]Finally, we will define our project's parameters by generating a project JSON file. This JSON file will contain a full description of the project to create with all parameters, layers to use, scenarios to create, scenario parameters…Instead of creating this file by hand, we will use thecreate_json_projectscript with options as follows:echoCreatingjsonproject
create_json_project\-o"systra"\-n"MyCity"\-g"$PROJECTS_DATA_PATH/mycity/mycity-gtfs.zip"\-wDATA\--ref-name"Base System"\-s"Scénario 1"allFalse\-lsurfaces_density\systrasaas.mycity_pop_2018\mycity_pop_2018\poppoppoppoppop\"$PROJECTS_DATA_PATH/mycity/mycity_population_2020_simplified.shp"\permis_den\permisDens\5,EFF3FF,20,B5CAE6,100,7BA2CD,500,4179B4,1700,08519C\--param-buffer\21000\3500\300\--param-typical-days\"JOB"20200121\"SAT"20200125\"SUN"20200119\--param-time-types\"Morning rush hour"070000095959\"Midday"100000155959\"Evening rush hour"160000192959\--param-center"6.342383""46.075223"\--param-distance"m"\--param-currency-symbol"€"You can find additional information on how to use this script by using the-hoption.All output files needed by ItSim's backend will be output in theDATAdirectory.Import a project into ItSimCongratulations! Now, that you have a valid GTFS, shapefiles and a project description JSON, you need to copy them in thesws_itsimbackend directory:[itsim-project-data] $ cp -r DATA/* ../sws_itsim/You are finally ready to import a project into ItSim!Go to yoursws_itsimdirectory and activate your Python virtual environment by executing the commandpipenv shell. Then, check that all the required files exist.We can now use the script./bootstrap_db/launcherto import the project. We will cover the essential options you will need below. In case you need more information on all the options and how to use the import script, it is strongly advised to use the-hoption.Example:./bootstrap_db/launcher -hFirst of all, we will have to know on which platform we will deploy the project and have the rights to do it. Indicate the backend's URL in the-uoption and the access token with the-toption. To use your token easily, you can store its value into an environment variable:export $ITSIM_TOKEN=[your token]. Note that your token must be a super-admin's token because only a super-admin has the right to create a project.In our case, we have a layer to display on the map, so we would like to useMapbox. To be able to connect to the service we also need a token. When you have the mapbox access token, you can store it into an environment variable as we did for ItSim's token:export $MAPBOX_TOKEN=[your mapbox token]Finally, we will have to indicate the project and the GTFS file's path with the-Poption as well as the organization with the-ooption.For our example, we will import the project on our local backend (on5001port):[sws_itsim] $ ./bootstrap_db/launcher -t "$ITSIM_TOKEN" -T "$MAPBOX_TOKEN" -u http://localhost:5001 -o systra -P mycity gtfs/mycity-gtfs.zipNote that you can check that your project description JSON is valid by adding the-noption at the end of the command. It will not do anything else and it is advised before trying to import a project for the first time.When it is all right, you can remove the-noption and execute the command. Then, confirm the operation by enteringyoryeswhen asked.