Package: pyshopify
Shopify Orders Rest API Wrapper and Data ExportpyShopify is a python wrapper for Shopify's REST API that pulls and inserts store data into a relational database and exports to a CSV file. This library is compatible with Microsoft SQL Server, MariaDB, and MySQL.This has the flexibility of exporting the data to MS SQL Server or MySQL (and MariaDB), exporting to csv and yielding or returning a dictionary of dataframes.The Structure of the Database is described in depth in theDocsfolder. The database can be built through theshopify_dbCLI command or programmatically.This requires the creation of a private app in the Shopify Admin center -> Setting -> Apps and Sales Channels -> Develop Apps.Table of ContentsShopify Orders Rest API Wrapper and Data ExportTable of ContentsData ReturnedOrders Datacustomers DataProduct DataConfigurationDefault ConfigurationEnvironment VariablesExampleconfig.iniConfigurationconfig.iniSectionsMicrosoft SQL Serverconfig.iniSectionConfiguration DictionaryInstallationSQL Driver RequirementsInstalling MySQLdb (MySQLClient) DriverInstalling pyMySQL DriverInstalling MS SQL ODBC DriverInstalling Python and Virtual EnvironmentCreating the Database and TablesCLI CommandProgramatically via PythonRun pyShopify with in Python ScriptInstantiating theShopifyAppClassUpdating ConfigurationProduct and Inventory DataMethods that Return products and Inventory DataGet All Product, Variant and Product Option DataFramesGet Inventory Levels and Locations DetailsMethods that Write Inventory and Product DataOrders and customers DataReturning/Yielding customers & Orders DataWriting Orders and Customers DataWriting Data to Desired OutputWriting All Data to DB or CSVRunning as Command Line ApplicationData StructureDocker ContainerDocker ComposeMS SQL Docker ContainerMySQL Docker ContainerCreating a ContainerRunning pyshopify in the containerData ReturnedThe following is a brief summary of the data being returned. The keys of the return dictionary match the table names of the database structure. The DataFrame columns match the columns of each table. For a detailed view of the structure of each DataFrame, see the database documentation.Orders DataKey/TableDescriptionordersOrder ID's, date, pricing details, fulfilment status, financial statusorder_attrAttribution details of each order - landing page, source url, referral url, campaignorder_pricesPrices, discounts and charges for each order.line_itemsOrder line items with order ID, date, quantity, product/variant ID's, sku, title, pricingadjustmentsOrder Refund adjustmentsdiscount_appsDiscount Applications for Each Orderdiscount_codesDiscount Codes In Use for Each Orderrefund_line_itemRefunded UnitsrefundsOrder refundsship_linesOrder ID, shipping pricing, carrier, code, source, titlecustomers DataKey/TableDescriptioncustomersCustomer purchase totals, numbers of orders, last order, geographyProduct DataNameDescriptionproductsProduct & Listing DatavariantsProduct Variant details & Inventoryproduct_optionsProduct & variant options configurationsinventory_levelsInventory Levels for items at each inventory locationinventory_locationsInventory locationsConfigurationThe application can be configured three ways, starting with the lowest priority:Default configuration - loaded automaticallyReading Environment Variables. When using the included docker container, these environment variables are automatically set from docker compose.File namedconfig.iniin current working directory or with relative or absolute file path passed in as an argumentapp = pyshopify.ShopifyApp(config_dir='/home/user/app/config.ini').Configuration dictionary with 'shopify', 'sql' and/or 'csv' keys as the ini file is structured.Default ConfigurationNoteSchema is based on the MS SQL Server construct, not MySQL. MySQL does not have schemas in the same sense as MSSQL, therefore it will have no impact if defined for a MySQL database.This is the default configuration automatically passed toShopifyAppinstance.default_configuration={'shopify':{'items_per_page':250,'days':7,'admin_ep':'/admin/api/','customers_ep':'customers.json','orders_ep':'orders.json','version':'2022-07','time_zone':'America/New_York',},'sql':{'connector':'mssql+pyodbc','azure':False,'encrypt':False,'windows_auth':False,'db':'shop_rest'},'csv':{'filepath':'csv_export',},}Environment Variables# Shopify Environment Variables$exportSHOPIFY_ACCESS_TOKEN=<youraccesstoken>
$exportSHOPIFY_STORE_NAME=<yourstorename>
$exportSHOPIFY_API_VERSION=<Shopifyapiversion># ex. 2022-07# SQL Server Environment Variables$exportSHOPIFY_DB_USER=<yourshopifydbuser>
$exportSHOPIFY_DB_PASSWORD=<yourshopifydbpassword>
$exportSHOPIFY_DB_NAME=<yourshopifydbname>
$exportSHOPIFY_DB_SERVER=<yourdbserver>
$exportSHOPIFY_DB_SCHEMA=<yourschemaname>Exampleconfig.iniConfigurationThe python script is configured through an ini based file usingconfigparser.config.iniSectionsThis is the structure of theconfig.inifile. The sections areshopify,sqlandcsv. Theshopifyandsqlsections are required, the andcsvsection is optional.The date range to pull the data can be set to the last number of days usingdaysor a specific date range usingstartandend. If both are set,start_endandend_datewill take precedence.[shopify]# Optional Configurationorders_ep=orders.jsoncustomers_ep=customers.jsonapi_version=2022-07api_path=/admin/api/items_per_page=250# Required Configurationstore_name=***STORE-NAME-IN-ADMIN-URL***# Shopify Access Tokenaccess_token=***ACCESS TOKEN FROM ADMIN***# Uncomment to get data between two dates# Overrides past days of history - Can be in 'YYYYMMDD' or 'MM-DD-YYYY' format# start = 20210610# end = 20210617# Get past days of history - 7 days is defaultdays=30[csv]# Relative directory for CSV exportsfilepath=csv_export[sql]# Configuration for Database - See sections belowMicrosoft SQL Serverconfig.iniSectionThe following is an example ofconfig.inisection for Microsoft SQL Server. The connector ismssql+pyodbc. To use windows authentication, setwindows_authto True, it defaults to False. Thedbis the database of tables. If no schema is set,dbowill be used.portis optional, if not set, the default port for the connector will be used.If using Azure MySQL or SQL Server, setazure = trueandencrypt = true. If using a self-signed certificate, setssl_certto the file path or url of the certificate. Thessl_certvalue is ignored for MSSQL, the certificate must be installed in the system store.TrustedServerCertificateis required to beTruewith theencryptoption, unless you have a valid certificate matching your domain from a CA.Please noteIf running this from a local computer, the firewall rule must be set on Azure to allow access to your current IP address.[shopify]......[sql]# Set true if using Azure serverazure=false# DB dialect & driver - first part of connection string -eg mssql-pyodbc, mysql-pymysqlconnector=mssql+pyodbc# Set to True to enable ssl encryption (usually required for Azure)encrypt=false# Option valid only for MSSQL - Do not verify server certificateTrustedServerCertificate=yes# Set MSSQL Driver Name - Optional, driver is automatically detected; driver = ODBC Driver 17 for SQL Server# Connection string query for other connection parameters# use comma separated list, i.e. param1:value1, param2:value2# DO NOT include any of the configuration options set above; connetion_query =# Database server & port - localhost for docker containerserver=***SERVER_ADDRESS***port=1433database=shop_rest# Schema is optional, based on MSSQL construct (no impact for MySQL)schema=dbo# Database user & passworddb_user=sadb_pass=***DATABASE_PW***MySQL ConfigurationBothmysql+pymysqlandmysql+mysqldbare supported. This is the new version of mysqldb -MySQLClient. Other drivers have not been tested.If connecting to azure, setazure = trueandencrypt = trueif required to use SSL/TLS. The currentDigiCertcertificate will be automatically downloaded from Microsoft unlessssl_certis set.[shopify]......[csv]...[sql]# Set true if using Azure serverazure=false# DB dialect & driver - first part of connection string -eg mssql-pyodbc, mysql-pymysqlconnector=mysql+pymysql# Set to True to enable ssl encryption (usually required for Azure)encrypt=false# Database server & port - localhost for docker containerserver=***SERVER_ADDRESS***# The default SSL certificate is automatically loaded for Azure MySQL# if using another replace this value with file path or url# Has no impact on MSSQL - CA cert must be in system store; ssl_cert = https://cacerts.digicert.com/DigiCertGlobalRootG2.crt.pemport=1433database=shop_rest# Database user & passworddb_user=***DB_USER***db_pass=***DATABASE_PW***# Connection string query for other connection parameters# use comma separated list, i.e. param1:value1, param2:value2# DO NOT include any of the configuration options set above; connetion_query =Configuration DictionaryThe configuration dictionary overrides theconfig.inifile. It follows the same structure of theconfig.inifile.frompyshopify.runnerimportShopifyAppconfiguration={"shopify":{"access_token":"<your api key>","store_name":"<your store name>","api_version":"2022-07","days":7,},"sql":{"connector":"mssql+pyodbc","azure":False,"encrypt":False,"TrustedServerCertificate":True,"db":"**DB_NAME**","server":"**SERVER_ADDRESS**","port":1433,"db_user":"**DB_USER**","db_pass":"**DB_PASS**","schema":"dbo",},"csv":{"filepath":"csv_export",}}app=ShopifyApp(config_dict=configuration)InstallationSQL Driver RequirementsBefore pyShopify can be installed, the appropriate requirements for the SQL driver must be installed. The following drivers are supported:pyodbc- SQL Server Driver, requires additional installation steps.pymysql- Pure python MySQL driver, the easiest to install, no prerequisites needed.mysqldb- New fork of C-based MySQLdb driver. Faster than pymysql but requires additional installation steps.Installing MySQLdb (MySQLClient) DriverThe MySQLdb driver (MySQLClient) requires MySQL C libraries if building from source. On Windows, theMariaDB C Connectermay be required.On Ubuntu, the following packages should be first installed:aptinstall-ybuild-essentiallibssl-devpython3-devlibmysqlclient-devThen install the python package:python3-mpipinstallmysqlclientThe python package can also be installed with the pyshopify package:pip install pyshopify[mysqldb]Installing pyMySQL DriverIt's pure python, so it's as simple as:pipinstallpymysqlNote: you can also use the pip extras to install the python packagepip install pyshopify[pymysql]Installing MS SQL ODBC DriverOn linux, msodbc18 needs to be installed first, then unixodbc headers. This is an example on Ubuntu (only compatible with 18.04, 20.04 & 21.04):$sudosu# Install curl and gnupg2 if needed$aptupdate&&aptinstallcurlgnupg2lsb_release# Add Microsoft's Repository$curlhttps://packages.microsoft.com/keys/microsoft.asc|apt-keyadd-
$curlhttps://packages.microsoft.com/config/ubuntu/$(lsb_release-rs)/prod.list>/etc/apt/sources.list.d/mssql-release.list# Install ODBC Driver and unixodbc-dev headers$aptupdate&&aptinstall-ymsodbcsql18unixodbc-devThe Windows ODBC driver can be found hereMicrosoft SQL OBC DriverInstalling Python and Virtual EnvironmentOn Ubuntu or WSL:# Install python & venv if needed$aptinstallpython3.9python3.9-devpython3-pippython3-venvgit
$exit# exit sudo$cd~
$python3.9-mvenv~/venv
$gitclonehttps://github.com/webdjoe/pyshopify.git
$source~/venv/bin/activate&&pipinstall-r./pyshopify/requirements.txtCreating the Database and TablesThe database and table structure can be created one of two ways - CLI command or via Python script. All have the ability to create a new database or use an existing one. A new user can also be created with exclusive access to that database. It is recommended to create a new user instead of using the admin user when running the script. A new schema can also be created in MSSQL or the defaultdboschema can be used.CLI CommandUse theshopify_dbcommand to create the database, user and tables.shopify_db--help

Usage:shopify_db[OPTIONS]pyshopifydatabasebuilder.Buildonexistingdatabasewith--tables-onlyNewusercanbecreatedwith--db-userand--db-passForMSSQL,--schemawillbecreatedifitdoesnotexistspecifyingdate_startwillbuildadatedimensiontablestartingatdate_start.Useflag--azureand--encrypttoconnecttoAzureSQL/MySQLserver

Options:--db-tables/--tables-onlySpecifywhethertobuilddatabaseortablesonly-c,connectorTEXTDatabaseconnector-mysql+mysqldb,mysql+pymysql,mssql+pyodbc-s,--serverTEXTSQLServernameorIPaddress--azureEnableAzureSQLServerconnection--encrypt/--no-encryptEnableSSLencryption[default:False]-p,--portINTEGERSQLServerport-U,--sa-userTEXTSQLServerSAuser-W,--sa-passTEXTSQLServerSApassword-d,--db-nameTEXTDBtocreateorexistingdb--schemaTEXTSchematocreateorexistingschema--db-userTEXTDBusertocreate--db-passTEXTCreatedDBuserpassword--driverTEXTSQLServerODBCdriver--date-startTEXTCreateDateDimtablefromstartingdateYYYY-MM-DD--helpShowthismessageandexit.Build a new MSSQL database, user and schema with the following command:shopify_db-cmssql+pyodbc-slocalhost-p1433\-Usa-WTheStrongestpassword1!-dtest_shop\--db-usertest_shop_user--db-passTestShopPass1!\--schemaNewSchema--date-start2015-01-01Build tables into existing MSSQL database, no new user or schema:shopify_db--tables-only-cmssql+pyodbc-slocalhost-p1433\-Usa-WTheStrongestpassword1!-dtest_shop\--date-start2015-01-01Build a new MySQL database and user with the following command:shopify_db-cmysql+pymysql-slocalhost-p3306\-Usa-WTheStrongestpassword1!-dtest_shop\--db-usertest_shop_user--db-passTestShopPass1!\Create user and tables in existing Azure SQL Databaseshopify_db--tables-only-cmssql+pyodbc-sazure.database.windows.net\--azure--encrypt-p3306-Usa-WTheStrongestpassword1!\-dtest_shop--db-usertest_shop_user--db-passTestShopPass1!\Programatically via PythonTheDBFactoryclass in thepyshopify.sqlmodule can be used to create the database and tables, as well as user and/or schema.frompyshopify.sqlimportDBFactory# port, schema, driver and connect_query are optional for MSSQL# Schema and driver have no impact on MySQLdb_builder=DBFactory(connector='mssql+pyodbc',server=localhost,port=1433,db_name='shop_rest',sa_user='sa',sa_pass='password',schema='Shopify',driver='ODBC Driver 17 for SQL Server',azure=False,encrypt=False,windows_auth=False)# Create databasedb_builder.create_db()# Optional but recommended, create a new login and user with appropriate permissionsdb_builder.create_user('shopify_user','password')# Build tables and the new schema if specifieddb_builder.create_tables()# Optional, Build DateDimension tabledb_builder.create_date_dimension('2015-01-01')Run pyShopify with in Python ScriptThe primary class ispyshopify.runner.ShopifyApp()which contains the necessary execution methods.Instantiating theShopifyAppClassInstantiateShopifyAppfrom thepyshopify.runnermodule.frompyshopify.runnerimportShopifyApp# Instantiate ShopifyApp() with a configuration dictionary or absolute or relative location of config.ini fileshop_instance=ShopifyApp(config_dir='rel/dir/to/config.ini')# Use configuration dictionary above to instantiate ShopifyApp()shop_instance=ShopifyApp(config_dict=configuration)Updating ConfigurationThe configuration can be updated at any point with theupdate_config()method. This can update the dates or time period of the export. Any date periods specified will override the current configuration, regardless of precedence.# A full or partial configuration dictionary can be passed to update_config()updated_config={'shopify':{'start':'2020-01-01','end':'2020-01-31'}}shop_instance.update_config(updated_config)Product and Inventory DataProduct and variant data can be pulled into a dictionary of DataFrames or written to SQL Server or a CSV file.Methods that Return products and Inventory DataThese methods retrieve inventory locations, inventory levels and product details into a dictionary of DataFrames. The dictionary keys are the respective table names with the same columns.Get Inventory Levels at all locationsTheinventory_table()method returns a single dataframe with the product and variant id and title at all inventory locations. The method accepts lists or comma separated strings of product ids, variant ids and location_id. If no ids are specified, all products and variants will be returned.frompyshopify.runnerimportShopifyApp()app=ShopifyApp()inventory_levels=app.inventory_table(product_ids=[],variant_ids=[],location_ids=[])# Returns DataFrame of inventory items with product & variant id & title# Returns:# DataFrame(columns=[# 'product_id', 'variant_id', 'sku', 'product_title', 'variant_title',# 'available', 'location_id'])Get All Products and Inventory Detail DataFramesTheget_products_inventory()method returns a dictionary of DataFrames with product, variant, product option, inventory locations & inventory levels. The dictionary keys are the respective table names with values containing a dataframe of the associated table structure.product_details=app.get_products_inventory()product_details.items()# Returns:#  [(products DataFrame)#  (variants, DataFrame),#  (product_options, DataFrame),#  (inventory_locations, DataFrame),#  (inventory_levels, DataFrame)]Get All Product, Variant and Product Option DataFramesTheget_products()method returns a dictionary of DataFrames containing the product, variant and product option data. The dictionary keys are the respective table names with values containing a dataframe of the associated table structure.The variants table contains the quantity of product that is available accross all locations.product_details=app.get_products()product_details.items()# Returns:#  [(products DataFrame)#  (variants, DataFrame),#  (product_options, DataFrame)]Get Inventory Levels and Locations DetailsTheget_inventory_levels()method returns a dictionary of DataFrames containing the inventory levels and inventory locations data. The dictionary keys are the respective table names with values containing a dataframe of the associated table structure.A list or comma separated string of location ids can be passed to filter the inventory levels to the specified locations.# Get inventory levels at specific or all locations# Locations can be passed as a list or comma separated string of location ID's# Get location ID's from the get_inventory_locations() methodinventory_levels=app.get_inventory_levels(locations=[])inventory_levels.items()# Returns:#  [(inventory_levels, DataFrame)]Theget_inventory_locations()method returns a dictionary containing a DataFrame of inventory locations.# Get Dictionary with DataFrame of locations inventory is being heldinv_locations=app.get_inventory_locations()inv_locations# Returns:# {"inventory_locations": DataFrame(# columns=['id', 'name', 'address1', 'address2',#          'city', 'province', 'country', 'zip',#          'phone', 'active', 'updated_at'])}Methods that Write Inventory and Product DataThe following functions output the inventory and products data to a CSV file or SQL Server. The CSV file will be written to the directory specified in the configuration.A configuration dictionary can be passed to each method to override the current configuration. The configuration dictionary can be a full or partial configuration dictionary.# Write All products and All Inventory Data to CSV or SQL Serverapp.products_inventory_writer(write_sql=True,write_csv=Tru,config_dict=None)# Write products, variants & options to CSV or SQL Serverapp.products_writer(config_dict=None)# Write inventory levels to CSV or SQLapp.inventory_levels_writer(write_csv=True,write_sql=True,config_dict=None)# Write inventory locations to CSV or SQLapp.inventory_locations_writer(write_csv=True,write_sql=True,config_dict=None)Orders and customers DataThe following methods return the order details, pricing information and customers data. The methods return or yield a dictionary of DataFrames. The dictionary keys are the respective table names with the same columns. See the database documention for full column types and descriptions.Returning/Yielding customers & Orders DataThe customers and orders data can be returned or yielded, in addition to written to the database or CSV file. A full or partial configuration dictionary can be passed to these methods to override the current configuration.All of these methods can accept a configuration dictionary with just the shopify section.Shopify Configuration DictionaryThe shopify configuration dictionary is the sql section ofconfig.inishop_config={'start':'2020-06-01','end':'2020-06-30'}Get Full Customers and Orders DataFramesThis method will return a full dataset of customers and orders DataFrames for the specified time period. The method returns a dictionary of DataFrames with the dictionary keys being the respective table names.These methods are not recommended for large time periods due to memory constraints.frompyshopify.runnerimportShopifyAppapp=ShopifyApp()shopify_config={"access_token":"<your api key>","store_name":"<your store name>","api_version":"2022-07","days":7}# Return full customers and orders datasetorders=app.get_orders_customers_df(shopify_config=shopify_config)orders.items()# Returns:#   [('customers', <pandas.DataFrame>),#       ('Orders', <pandas.DataFrame>),#       ('refunds', <pandas.DataFrame>),#       ('line_items', <pandas.DataFrame>),#       ('refund_line_item', <pandas.DataFrame>),#       ('adjustments', <pandas.DataFrame>),#       ('discount_apps', <pandas.DataFrame>),#       ('discount_codes', <pandas.DataFrame>),#       ('ship_lines', <pandas.DataFrame>),#       ('order_prices', <pandas.DataFrame>),#       ('order_attr', <pandas.DataFrame>)]# Return full customers datasetcustomers=app.get_full_customers_df()customers.items()# Returns:#   [('customers', <pandas.DataFrame>)]# Return full orders and customers datasetall_data=app.get_full_orders_df(shopify_config=shopify_config)all_data.items()# Returns:#   [('Orders', <pandas.DataFrame>),#       ('refunds', <pandas.DataFrame>),#       ('line_items', <pandas.DataFrame>),#       ('refund_line_item', <pandas.DataFrame>),#       ('adjustments', <pandas.DataFrame>),#       ('discount_apps', <pandas.DataFrame>),#       ('discount_codes', <pandas.DataFrame>),#       ('ship_lines', <pandas.DataFrame>),#       ('order_prices', <pandas.DataFrame>),#       ('order_attr', <pandas.DataFrame>)]Iterating through Orders and customers DataFor larger datasets, it is recommended to use the generator methods to iterate through the paginated data. The methods yield the same structure as theget_full_orders_df()andget_full_customers_df()methods.Ashopify configuration dictionarycan be passed to these methods as well.frompyshopify.runnerimportShopifyAppapp=ShopifyApp()# A shopify configuration dictionary can be passed to all the generator methodsfororder_dictinapp.orders_iterator(shopify_config=shopify_config):order_dict.items()# Returns:#   [('Orders', <pandas.DataFrame>),#       ('refunds', <pandas.DataFrame>),#       ('line_items', <pandas.DataFrame>),#       ('refund_line_item', <pandas.DataFrame>),#       ('adjustments', <pandas.DataFrame>),#       ('discount_apps', <pandas.DataFrame>),#       ('discount_codes', <pandas.DataFrame>),#       ('ship_lines', <pandas.DataFrame>),#       ('order_attr', <pandas.DataFrame>)]#       ('order_prices', <pandas.DataFrame>)]# Iterate through customers dataforcustomers_dictinapp.customers_iterator():customers_dict.items()# Returns:#   [('customers', <pandas.DataFrame>)]Writing Orders and Customers DataThere are several convenience methods that can be used to write the orders and customers data to CSV and SQL Server. The sql configuration can be passed directly to these methods, as theengineis only created during the first SQL method and stored as an instance variableapp.engineWriting Data to Desired Outputfrompyshopify.runnerimportShopifyApp# No arguments defaults to attempt to read to a config.ini file in CWDapp=ShopifyApp()# Main method that can write order & customer data to csv/sql and set configuration. A full configuration dictionary can be passed to this method.app.orders_customers_writer(write_csv=True,write_sql=True,config=full_config)# Write only orders data to CSV and/or SQLapp.orders_writer(write_sql=True,write_csv=True,config=full_config)# Write only customers data to CsV and/or SQLapp.customers_writer(write_sql=True,write_csv=True,config=full_config)Writing All Data to DB or CSVA convenience method can be used to write all data to the database or CSV. This accepts a full configuration dictionary.frompyshopify.runnerimportShopifyAppapp=ShopifyApp()# Write product, inventory, customers, and orders data to dbapp.write_all_to_sql(config=full_config)# Write product, inventory, customers, and orders data to csvapp.write_all_to_csv(config=full_config)Running as Command Line ApplicationThe command line applicationshopify_clican output the data to CSV or SQL Server. The command line application can be run with the following command:From command line:# There are different command options$shopify_cli--help
Usage:shopify_cli[OPTIONS]RunShopifyAppCLI.Use-dor--daystogetdaysofhistory,default7Use-bor--betweentogetdatabetween2dates-yyyy-MM-ddyyyy-MM-dd,--configistherelativeorabsolutelocationofconfig.ini

Options:--allGetallproduct,orderandcustomerdata--ordersGetordersdata--customersGetcustomersdata--productsGetproductsdata-d,--daysINTEGERgetdaysofhistory,default7-b,--betweenTEXT...getbetween2dates-yyyy-MM-dd,ex-b2020-01-012020-01-02--sql-out/--no-sqlwritetodatabase-DefaultFalse--csv-out/--no-csvWriteresultstocsvfiles-Defaulttrue--csv-locationTEXTRelativefolderofcsvexportfolderdefaultstocsv_export/--configTEXTRelativelocationofconfig.ini-defaultstoconfig.iniincurrectdirectory# Write last 30 days of all data to DB$shopify_cli--all-d30--sql-out--configconfig.ini# Write 30 days of only orders data to CSV$shopify_cli--orders-d30--csv-out--configconfig.ini# Get customers created between 2 dates$shopify_cli--customers-b2020-01-012020-01-02--sql-outData StructureThe structure of the custom return dictionary reflects the SQL database structure that it will update. See thedatabase docsfor full details on each table/DataFrame.frompyshopifyimportShopifyApp# Enable custom in config.inishop_app=ShopifyApp()orders_dict=shop_app.orders_full_df()orders_dict={'Orders':DataFrame,'refunds':DataFrame,'line_items':DataFrame,'refund_line_item':DataFrame,'adjustments':DataFrame,'discount_apps':DataFrame,'discount_codes':DataFrame,'ship_lines':DataFrame,'order_attr':DataFrame,'order_prices':DataFrame}customers_dict=shop_app.customers_full_df()customers_dict={"customers":DataFrame}Docker ContainerThere are two docker containers that can be used to run the application - an MS SQL based container and a MySQL based container. They are located in the respective docker folders,docker-mysqlanddocker-mssql. Set the environment variables in the docker-compose.yml file before running.Both containers have the same structure and can be used interchangeably. The only difference is the database engine used.Set environment variables and SA password to configure the database indocker-compose.yml. Configure a volume for CSV export if neededTheconfig.inishould be mapped to the/rootuser directory inside the container. A csv folder can also be mapped to a local folder. This should match the relative csv filepath in the configuration file. The working directory is the root user's home directory.Docker ComposeThe docker compose file configures the db administrator and the shopify database user. The date dimension table is built based on theSTARTING_DATEenvironment variable.The default location of theconfig.inifile is in the root user dir, the container's working directory. The data directory volume is the db data folder and the csv_export folder can also be mapped to a local directory.MS SQL Docker ContainerThe Microsoft SQL Server container is based on the Microsoft image with Ubuntu 20.04 and the pyshopify library installed alongside python 3.9. The container is configured to run theshopify_clicommand line application.On the initial run, the database structure is automatically built based on the environment variables in docker compose. Theconfig.inifile is automatically populated with the database details based on the environment variables.services:sqlserver:build:context:..dockerfile:docker-mssql/Dockerfilecontainer_name:shop_sqlhostname:shop_sqlports:-"1433:1433"environment:ACCEPT_EULA:"Y"SA_PASSWORD:"TheStrongestpassword1!"MSSQL_SA_PASSWORD:"TheStrongestpassword1!"SHOPIFY_DB_USER:"shop_user"SHOPIFY_DB_PASSWORD:"Strongerpassword1!"SHOPIFY_DB_NAME:"shop_rest"SHOPIFY_DB_SCHEMA:"dbo"STARTING_DATE:"2019-01-01"CONFIG_FILE:"/root/config.ini"volumes:-"../csv_export:/root/csv_export"-"../data:/var/opt/mssql"-"../config.ini:/root/config.ini"MySQL Docker ContainerThe MySQL container is based on the MySQL image with Debian-8 and the pyshopify library installed alongside python 3.9. The container is configured to run theshopify_clicommand line application.On the initial run, the database structure is automatically built based on the environment variables in docker compose. Theconfig.inifile is automatically populated with the database details based on the environment variables.services:
    mysql-db:
        build:
            context: ..
            dockerfile: docker-mysql/Dockerfile
        container_name: shop_mysql
        hostname: shop_mysql
        ports:
            - "3306:3306"
        environment:
            MYSQL_TCP_PORT: 3306
            MYSQL_ROOT_PASSWORD: StrongestPassword1
            DB_NAME: shop_sql
            DB_USER: shop_user
            DB_PASS: StrongerPassword1
            STARTING_DATE: "2019-01-01"
            CONFIG_FILE: "/root/config.ini"
        volumes:
            - "../csv_export:/root/csv_export"
            - "../data:/var/lib/mysql"
            - "../config.ini:/root/config.ini"Creating a ContainerTo create a container:gitclonehttps://github.com/webdjoe/pyshopify# Switch to the appropriate docker container foldercdpyshopify/docker-mssqlcdpyshopify/docker-mysql# edit the docker compose environment variables with vim or nanovimdocker-compose.yml# Build and run container detachedsudodockercomposeup--build-d# edit config.ini filevimconfig.iniOnce started test if SQL server is running. A0return value indicates the server has started up and is ready for connections.sudodockerexec-itshop_sqlbash-c'/opt/mssql-tools/bin/sqlcmd -h -1 -t 1 -U sa \-P "$MSSQL_SA_PASSWORD" -Q "SET NOCOUNT ON; Select SUM(state) from sys.databases"'Running pyshopify in the containershopify_clican be called in container to pull data and send to database or csv files. Check out theshopify_clisection for more details.# Get last 30 days of data and import into SQL Server running in containersudodockerexec-itshop_sqlshopify_cli--all-d30--sql-out# Get orders data between dates and import into SQL Serversudodockerexec-itshop_sqlshopify_cli--orders-b2020-01-012020-12-31--sql-out# Pull 30 days of data into csv filessudodockerexec-itshop_sqlshopify_cli--csv-out--csv-locationcsv_export