Package: ugrc-palletjack
agrc/palletjackA library of classes and methods for automatically updating AGOL feature services with data from several different types of external sources. Client apps (sometimes called 'skids') can reuse these classes for common use cases. The code modules are oriented around each step in the extract, transform, and load process.palletjackworks with pandas DataFrames (either regular for tabular data or Esri's spatially-enabled dataframes for spatial data). The extract and transform methods return dataframes and the load methods consume dataframes as their source data.Thedocumentationincludes a user guide along with an API description of the available classes and methods.Pallet jack:forklift'slittle brother.Dependenciespalletjackrelies on the dependencies listed insetup.py. These are all available on PyPI and can be installed in most environments, including Google Cloud Functions.Thearcgislibrary does all the heavy lifting for spatial data. If thearcpylibrary is not available (such as in a cloud function), it relies onshapelyfor its geometry engine.InstallationActivate your application's environmentpip install ugrc-palletjackQuick startImport the desired modulesUse a class inextractto load a dataframe from an external sourceTransform your dataframe as desired with helper methods fromtransformUse the dataframe to update a hosted feature service using the methods inloadfrompalletjackimportextract,transform,load#: Load the data from a Google Sheetgsheet_extractor=extract.GSheetLoader(path_to_service_account_json)sheet_df=gsheet_extractor.load_specific_worksheet_into_dataframe(sheet_id,'title of desired sheet',by_title=True)#: Convert the data to points using lat/long fields, clean for uploadingspatial_df=pd.DataFrame.spatial.from_xy(input_df,x_column='longitude',y_column='latitude')renamed_df=transform.DataCleaning.rename_dataframe_columns_for_agol(spatial_df)cleaned_df=transform.DataCleaning.switch_to_nullable_int(renamed_df,['an_int_field_with_null_values'])#: Truncate the existing feature service data and load the new datagis=arcgis.gis.GIS('my_agol_org_url','username','super-duper-secure-password')updates=palletjack.load.FeatureServiceUpdater.truncate_and_load_features(gis,'feature_service_item_id',cleaned_df,r'c:\directory\to\save\truncated\data\in\case\of\error')DevelopmentCreate a conda environment with Python 3.9conda create -n palletjack python=3.9activate palletjackClone the repoInstall in dev mode with development dependenciespip install -e .[tests]Troubleshooting Weird Append ErrorsIf aFeatureLayer.append()call (within a load.FeatureServiceUpdater method) fails with an "Unknown Error: 500" error or something like that, you can query the results to get more info. Theurllib3debug-level logs will include the HTTP GET or POST call, something like the following:https://services1.arcgis.com:443 POST /<unique string>/arcgis/rest/services/<feature layer name>/FeatureServer/<layer id>/append/jobs/<job guid>?f=json token=<crazy long token string>. The defualtbasicConfiglogger includes theurllib3logs (logging.basicConfig(level=logging.DEBUG)) and is great for development debugging, or you can add a specific logger forurllib3in your code and set it's level to debug.You can use this and a token from an AGOL tab to build a new job status url. To get the token, log into AGOL in a browser and open a private hosted feature layer item. Click the layer, and then open the developer console. With the Network tab of the console open, click on the "View" link for the service URL. You should see a document in the list whose name includes "?token=". Copy the name and then copy out the token string.Now that you've got the token string, you can build the status query:https://services1.arcgis.com/<unique string>/arcgis/rest/services/<feature layer name>/FeatureServer/<layer id>/append/jobs/<job guid>?f=json&<token from agol>Calling this URL in a browser should return a message that will hopefully give you more info as to why it failed.Updating Docspalletjack usespdoc3to generate HTML docs indocs/palletjackfrom the docstrings within the code itself. These are then served up via github pages.The github pages are served from thegh-pagesbranch. After you make edits to the code and update the docstrings, rebase this branch onto the updatedmainbranch. To prevent github pages from trying to generate a site from the contents ofdocs/palletjackwith jekyll, add a.nojekyllfile todocs/palletjack.To generate the docs, runpdoc --html -o docs\ c:\palletjack\repo\src\palletjack --force. The code's docstrings should be Google-style docstrings with proper indentation to ensure the argument lists, etc are parsed and displayed correctly.docs/README.mdis included at the top package level by adding the line.. include:: ../../docs/README.mdin__init__.py's docstring. This tells pdoc to insert that markdown into the HTML generated for that docstring, and the include directive can be used for more in-depth documentation anywhere else as well. Note thatpdoctries to create links for anything surrounded by backticks, which are also used for code blocks. You may need to manually edit the HTML to remove the links if they change the content of your code blocks (such as in the example import statement).Once the contents ofdocs/palletjacklook correct, force push thegh-pagesbranch to github. This will trigger the action to republish the site. The docs are then accessible at [agrc.github.io/palletjack/palletjack/index.html].