Package: dora-parser
ParserDora'sparserandtranspilertool for some big dataSQLdialects, based onMozilla SQL Parserproject.Getting StartedAn application that will translate source code from a given language (Impala,Spark,Hive,PrestoandAthena) and produce equivalent code in another language that has a similar level of abstraction.InstallationTo install dora-parser if you haspandasandseabornon your machine, usepipinstalldora-parserotherwise you can usepipinstalldora-parserpandasseabornUsageTranslateQueryI. Import Modulesfromdora_parser.parserimportParserfromdora_parser.transpilerimportTranspilerII. Generate the parsetreefromquerytree=Parser(query)Wherequeryhave the following value:WITHt1as(selectnow()att_day)SELECTDECODE(DAYOFWEEK(att_day),1,'Monday',2,'Tuesday',3,'Wednesday',4,'Thursday',5,'Friday',6,'Saturday',7,'Sunday','Unknown day')"Day of week",TRUNC(MONTHS_ADD(att_day,2),'SYEAR')as"Trunc SYEAR",TRUNC(MONTHS_ADD(att_day,2),'YEAR')as"Trunc YEAR"FROMt1;creates the tree object as shown below:{"select":[{"value":{"decode":[{"dayofweek":"att_day"},1,{"literal":"Monday"},2,{"literal":"Tuesday"},3,{"literal":"Wednesday"},4,{"literal":"Thursday"},5,{"literal":"Friday"},6,{"literal":"Saturday"},7,{"literal":"Sunday"},{"literal":"Unknown day"}]},"name":"Day of week"},{"value":{"trunc":[{"months_add":["att_day",2]},{"literal":"SYEAR"}]},"name":"Trunc SYEAR"},{"value":{"trunc":[{"months_add":["att_day",2]},{"literal":"YEAR"}]},"name":"Trunc YEAR"}],"from":"t1","with":{"name":"t1","value":{"select":{"value":{"now":{}},"name":"att_day"}}}}III. Translate thetreeobject from your original SQL Dialect (impala) to the new one (spark)transpiler=Transpiler(from_dialect='impala',to_dialect='spark')result,errors=transpiler.translate(tree)theresultvalue will be likeWITHt1AS(SELECTNOW()ASatt_day)SELECTCASEWHENDAYOFWEEK(att_day)=1THEN'Monday'WHENDAYOFWEEK(att_day)=2THEN'Tuesday'WHENDAYOFWEEK(att_day)=3THEN'Wednesday'WHENDAYOFWEEK(att_day)=4THEN'Thursday'WHENDAYOFWEEK(att_day)=5THEN'Friday'WHENDAYOFWEEK(att_day)=6THEN'Saturday'WHENDAYOFWEEK(att_day)=7THEN'Sunday'ELSE'Unknown day'ENDAS`Dayofweek`,TRUNC(att_day+INTERVAL2MONTHS,'SYEAR')AS`TruncSYEAR`,TRUNC(att_day+INTERVAL2MONTHS,'YEAR')AS`TruncYEAR`FROMt1You can also have access to a list with information about anyerrors, as well as where they occur.
In this example, TheTRUNCfunction inSparkonly works with a few data formats, so you can not use it with "SYEAR"[{"trunc":"MEDIUM:20:[{'add': ['att_day', {'interval': [2,'MONTHS']}]}, {'literal': 'SYEAR'}]"}]Information about the errors will also appear in the output log.dora_parser 2021-08-12 17:24:24,650 WARNING _TRUNC_ Spark 
data formats:['YEAR', 'YYYY', 'YY', 'QUARTER', 'MONTH', 'MM', 'MON', 'WEEK']
dora_parser 2021-08-12 17:24:24,650 WARNING resolve trunc NotImplemented: 
--TRANSPILER:MEDIUM:LEVEL20:'trunc'TranslateScriptAre considered anscriptany type ofstringwithmultiple SQL statementsI. Import Modulefromdora_parser.readerimportReaderII. Translatescript="""INSERT INTO t.customer SELECT DCEIL(p_sale) FROM Customers;COMPUTE STATS customer;SELECT staff_id, staff_name, CHAR_LENGTH(staff_name) AS lengthofname, COUNT(*) order_countFROM sales.ordersWHERE YEAR(order_date) = 2021GROUP BY staff_id;"""reader=Reader(from_dialect='impala',to_dialect='athena')result,errors,n_queries=reader.translate_script(script)Give you asresultINSERTINTOt.customerSELECTCEIL(p_sale)FROMCustomers;/* STATEMENT ERRORS:COMPUTE STATS*/COMPUTESTATScustomer;SELECTstaff_id,staff_name,LENGTH(staff_name)ASlengthofname,COUNT(*)ASorder_countFROMsales.ordersWHEREYEAR(order_date)=2021GROUPBYstaff_id;III. Generate a summary list (optional)summary=reader.create_summary(errors,n_queries)value forsummaryvariable:[{"N_queries":3},{"Success":2},{"Failed":{"HARD":1}},{"Er_types":["compute stats"]}]Translate multipleFilesI. Import Modulefromdora_parser.readerimportReaderII. Translatedir_impala='scrpits/impala/'dir_spark='scripts/spark'reader=Reader(from_dialect='impala',to_dialect='spark',input_dir=dir_impala,output_dir=dir_spark)reader.translate_files()The translated files will be saved to folders in the output directory according to the result of the translation.
If you don't specify the output directory, the resulting folders will be in the input directory.III. Generate a migration report (optional)You can also have access to a report in HTML with an overview of the result of migration process.
To do this, set themigration_reportargument equal to True".reader=Reader(from_dialect='impala',to_dialect='spark',input_dir=dir_impala,output_dir=dir_spark,migration_report=True)reader.translate_files()IV. Generate a summary (optional)If you want to access a summary dictionary of the migration process, set thesummary_dictargument equal to True, as in the example below:reader=Reader(from_dialect='impala',to_dialect='spark',input_dir=dir_impala,output_dir=dir_spark)reader.translate_files(summary_dict=True){"Input_dir":"/scripts/impala","From_dialect":"impala","To_dialect":"spark","Sucess_files":3,"Failed_files":1,"Files":{"row_benchmark.sql":[{"N_queries":7},{"Success":7},{"Failed":{}},{"Er_types":[]}],"customer_facts.sql":[{"N_queries":12},{"Success":12},{"Failed":{}},{"Er_types":[]}],"document.sh":[{"N_queries":1},{"Success":1},{"Failed":{}},{"Er_types":[]}],"orches.sql":[{"N_queries":9},{"Success":5},{"Failed":{"HARD":3}},{"Er_types":["create role","appx_median","parser"]}]}}Error TypesParser: Errors when generating the tree structure.Transpiler: Unimplemented functions and their levels of complexity.Statement: Commands that are not capable of being reproduced, since there are no equivalent in the target language.Supported LanguagesApache Impala 2.12Hive 3.12Spark 3.1Presto 0.217Getting HelpWe use GitHubissuesfor trackingbugs,questionsandfeature requests.ContributingPlease read through thiscontributingdocument to get start and before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your contribution.Dora Projectis a recent open-source project based on technology developed atCompasso UOL