Package: qapi-sdk
QAPI SDKQAPI SDK provides a library of classes for working with Query API in your Python code.RequirementsPython 3.6+Must be logged into the private VPN.Installationpipinstallqapi-sdkEnvironment VariablesQAPI_URL: QAPI Base URL.EMAIL: Your EmailOptional: If you choose to add your AWS credentials, you can use theread_columnsmethod to read in the
headers of your CSV file automatically.AWS_ACCESS_KEY_ID: AWS Access Key IDAWS_SECRET_ACCESS_KEY: AWS Secret Access KeyAWS_DEFAULT_REGION: AWS Default RegionExamplesQueryFEED ID: The table must exist in Athena.QUERY ID: The query id is used as an identifier for the query. Query id must be unique. Once you have retrieved your
data from S3 it is advised to delete the query.SQL: The SQL query to be executed.importtimefromdotenvimportload_dotenvfromqapi_sdkimportQueryload_dotenv()# Step 1: Assign your FEED ID, QUERY ID, and SQL QUERYfeed_id="[FEED/TABLE NAME]"query_id="[QUERY NAME]"query=f"SELECT * FROM{feed_id}"# Step 2: Create a Query objectmy_query=Query(feed_id=feed_id,query_id=query_id)# Step 3: Execute the query pushmy_query.push_query(sql=query)# Step 4: Wait for the query to completewhilemy_query.query_status():print("Waiting for query to complete...")time.sleep(10)# Step 5 (Optional): Delete the querymy_query.delete_query()FeedFEED ID: The table name you want to create in Athena.PUSH ID: The push id is used as an identifier for the query. Push id must be unique.COLUMNS: The name of the columns that will be pushed to Athena.importtimefromdotenvimportload_dotenvfromqapi_sdkimportFeedload_dotenv()# Step 1: Assign your FEED ID, PUSH ID, and COLUMNSfeed_id="[FEED/TABLE NAME]"push_id="[PUSH ID/PUSH NAME]"# Step 2: Create a Feed objectmy_feed=Feed(feed_id=feed_id,push_id=push_id)# Step 3: You can manually assign the columnscolumns=[{"name":"email","type":"string"},{"name":"md5email","type":"string"},{"name":"firstname","type":"string"}]# Step 3a (Optional): If you added AWS credentials, you can use the `read_columns` method to read# in the headers of your CSV file automatically.columns=my_feed.read_columns(data_bucket="[DATA BUCKET]",data_key="path/to/your/data/dir/ OR path/to/your/data/file.csv",delimiter=",")# Step 4: Define where to grab the data and format of the data.Then push the data to Athena.my_feed.push_feed(pull_path_bucket="[DATA BUCKET]",pull_path_key="path/to/your/data/dir OR path/to/your/data/file.csv",columns=columns,separator=",")# Step 5: Wait for the push to completewhilemy_feed.push_status():print("Waiting for push to complete...")time.sleep(10)# Step 6 (Optional): Delete the pushmy_feed.delete_push()# Step 7 (Optional): Delete the feedmy_feed.delete_feed()RedshiftFEED ID: You must use an existing feed.QUERY ID: The query id is used as an identifier for the query. Query id must be unique. Once you have retrieved your
data from S3 it is advised to delete the query.SQL: The SQL query to be executed.If you query an Athena table from Redshift, you must append the Athena schema to the table name.For example:SELECT * FROM [query_api].[TABLE NAME]If you use aLIMITclause, you must wrap the query in aSELECT * FROM ()clause.For example:SELECT * FROM (SELECT * FROM [TABLE NAME] LIMIT 100)importtimefromdotenvimportload_dotenvfromqapi_sdkimportRedshiftload_dotenv()# Step 1: Assign your FEED ID, QUERY ID, and SQL QUERYfeed_id="[EXISTING FEED ID]"query_id="[QUERY NAME]"query="SELECT * FROM (SELECT * FROM [SCHEMA].[TABLE NAME] LIMIT 10)"# Step 2: Create a Redshift objectmy_query=Redshift(feed_id=feed_id,query_id=query_id)# Step 3: Execute the query pushmy_query.push_query(sql=query)# Step 4: Wait for the query to completewhilemy_query.query_status():print("Waiting for query to complete...")time.sleep(10)# Step 5 (Optional): Delete the querymy_query.delete_query()CHANGELOG[0.3.4] - 2020-06-02UpdatedREADME.mdto inform the user they can push a directory of files with thepush_feedmethod or push a single
file with thepush_feedmethod.UpdatedREADME.mdto inform the user they can use theread_columnsmethod to read in the headers of your CSV
file from a directory or a single file automatically.Changed theread_columnsparameterdata_key_dirtodata_key. This is to make it consistent with the other
methods.Changed the order of operations steps in theREADME.mdfile to make it easier to follow.[0.3.3] - 2020-06-01Updated package to include Python 3.6+[0.3.2] - 2020-05-30UpdatedREADME.md[0.3.0] - 2020-05-30AddedRedshiftobject to the SDK.Addeddelete_querymethod to Redshift class.Addedquery_statusmethod to Redshift class.Addedpush_querymethod to Redshift class.UpdatedREADME.md[0.2.1] - 2020-05-30Addedhomepageandrepositorylinks to thepyproject.tomlfile.[0.2.0] - 2020-05-29AddedFEEDobject to the SDK.Addedread_columnsmethod to Feed class.Addeddelete_pushmethod to Feed class.Addeddelete_feedmethod to Feed class.Addedpush_statusmethod to Feed class.Addedpush_feedmethod to Feed class.UpdatedREADME.md[0.1.4] - 2022-05-29AddedQUERYobject to the SDK.Addeddelete_querymethod to Query class.Addedquery_statusmethod to Query class.Addedpush_querymethod to Query class.Added theCHANGELOGsection.UpdatedREADME.md