Package: outguncsv
Finds the right CSV separator and excludes bad linesExample:# You have probably seen this before, right?importpandasaspdpd.read_csv(r"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv")Traceback(mostrecentcalllast):File"C:\Users\Gamer\anaconda3\envs\dfdir\lib\site-packages\IPython\core\interactiveshell.py",line3398,inrun_codeexec(code_obj,self.user_global_ns,self.user_ns)File"<ipython-input-8-c5703c9ae399>",line1,in<cellline:1>....File"pandas\_libs\parsers.pyx",line1973,inpandas._libs.parsers.raise_parser_errorpandas.errors.ParserError:Errortokenizingdata.Cerror:Expected1fieldsinline3,saw10pd.read_csv(r"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv",on_bad_lines='skip')# Better, but everything in one column, and we have lost about 600 rows.SchweizerischeNährwertdatenbankkomplett(bearbeitetdurchLesleyGrünenfelder)(Stand:11.02.2023);;;;;;;;;;;;;;;;;;;;;;;;;;;SchweizerischeNährwertdatenbankkomplett(bearbeitetdurchLesleyGrünenfelder)(Stand:30.01.2023);;;;;;;;;;;;;;;;;;;;;0;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;1AgarAgar;;Gelier-undBindemittel;Verschiede...2Agavensirup;Agavendicksaft;ZuckerundSüsssto...3Ahornsirup;;ZuckerundSüssstoffe;Verschiedene...4Älplermagronen;zubereitet;salzigeGerichte;Ge........521Zwieback;;BrotundBackware;GetreideundGet...522Zwieback;Vollkorn;BrotundBackware;Getreide...523Zwiebel;gedünstet(ohneZugabevonFettundSa...524Zwiebel;geröstet(ohneZugabevonFettundSal...525Zwiebel;roh;Gemüsefrisch;Gemüse;pro100ges...[526rowsx1columns]# If you have problems reading a CSV file, use:fromoutguncsvimportread_balky_csv_filesalf3=r"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv"df3=read_balky_csv_files(csvfiles=alf3,encoding="utf-8",sep=None,regexremove=(),filepathcolumn="file",on_bad_lines="warn",)df3Out[4]:0...file0SchweizerischeNährwertdatenbankkomplett(bea......https://github.com/zdavatz/diprela/raw/main/cs...1NaN...https://github.com/zdavatz/diprela/raw/main/cs...2Name...https://github.com/zdavatz/diprela/raw/main/cs...3AgarAgar...https://github.com/zdavatz/diprela/raw/main/cs...4Agavensirup...https://github.com/zdavatz/diprela/raw/main/cs............1213Zwiebel...https://github.com/zdavatz/diprela/raw/main/cs...1214Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...1215Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...1216Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...1217Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...[1218rowsx50columns]fromoutguncsvimportread_balky_csv_filesimportglobalf=glob.glob(r"C:\Users\Gamer\Documents\Downloads\anyascii-master\input\tables\*.tsv")df=read_balky_csv_files(csvfiles=alf,# list or string (url/file path)encoding="utf-8",sep=None,# if None, it does its best to find the best separatorregexremove=(),# remove lines, regex must be in binary: rb"^\s*#\s+.*$",filepathcolumn="file",# a new colum will be created with the file pathon_bad_lines="skip",# use either skip or warn, it won't work with error# for *args, **kwargs -> https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html)alf=r"https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv"df2=read_balky_csv_files(csvfiles=alf,encoding="utf-8",sep=None,regexremove=(),filepathcolumn="file",on_bad_lines="skip",)alf3=r"https://github.com/zdavatz/diprela/raw/main/csv/diprela.csv"df3=read_balky_csv_files(csvfiles=alf3,encoding="utf-8",sep=None,regexremove=(),filepathcolumn="file",on_bad_lines="warn",)alf4="https://github.com/curran/data/raw/gh-pages/migrants/events.csv"df4=read_balky_csv_files(csvfiles=alf4,encoding="utf-8",sep=",",regexremove=(),filepathcolumn="file",on_bad_lines="skip",)dfOut[3]:01file0𞤢aC:\Users\Gamer\Documents\Downloads\anyascii-ma...1𞤣dC:\Users\Gamer\Documents\Downloads\anyascii-ma...2𞤤lC:\Users\Gamer\Documents\Downloads\anyascii-ma...3𞤥mC:\Users\Gamer\Documents\Downloads\anyascii-ma...4𞤦bC:\Users\Gamer\Documents\Downloads\anyascii-ma..........14439𜾿-C:\Users\Gamer\Documents\Downloads\anyascii-ma...14440𜿀-C:\Users\Gamer\Documents\Downloads\anyascii-ma...14441𜿁-C:\Users\Gamer\Documents\Downloads\anyascii-ma...14442𜿂-C:\Users\Gamer\Documents\Downloads\anyascii-ma...14443𜿃-C:\Users\Gamer\Documents\Downloads\anyascii-ma...[14444rowsx3columns]df2Out[4]:012...1011file0103...NaNShttps://raw.githubusercontent.com/pandas-dev/p...1211...C85Chttps://raw.githubusercontent.com/pandas-dev/p...2313...NaNShttps://raw.githubusercontent.com/pandas-dev/p...3411...C123Shttps://raw.githubusercontent.com/pandas-dev/p...4503...NaNShttps://raw.githubusercontent.com/pandas-dev/p.......................88688702...NaNShttps://raw.githubusercontent.com/pandas-dev/p...88788811...B42Shttps://raw.githubusercontent.com/pandas-dev/p...88888903...NaNShttps://raw.githubusercontent.com/pandas-dev/p...88989011...C148Chttps://raw.githubusercontent.com/pandas-dev/p...89089103...NaNQhttps://raw.githubusercontent.com/pandas-dev/p...[891rowsx13columns]df3Out[5]:0...file0SchweizerischeNährwertdatenbankkomplett(bea......https://github.com/zdavatz/diprela/raw/main/cs...1NaN...https://github.com/zdavatz/diprela/raw/main/cs...2Name...https://github.com/zdavatz/diprela/raw/main/cs...3AgarAgar...https://github.com/zdavatz/diprela/raw/main/cs...4Agavensirup...https://github.com/zdavatz/diprela/raw/main/cs............1213Zwiebel...https://github.com/zdavatz/diprela/raw/main/cs...1214Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...1215Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...1216Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...1217Zwiebelkuchen...https://github.com/zdavatz/diprela/raw/main/cs...[1218rowsx50columns]df4Out[6]:0...file0NaN...https://github.com/curran/data/raw/gh-pages/mi...157234.0...https://github.com/curran/data/raw/gh-pages/mi...256633.0...https://github.com/curran/data/raw/gh-pages/mi...372740.0...https://github.com/curran/data/raw/gh-pages/mi...455194.0...https://github.com/curran/data/raw/gh-pages/mi..............95536496.0...https://github.com/curran/data/raw/gh-pages/mi...95636500.0...https://github.com/curran/data/raw/gh-pages/mi...95736499.0...https://github.com/curran/data/raw/gh-pages/mi...95836503.0...https://github.com/curran/data/raw/gh-pages/mi...95936502.0...https://github.com/curran/data/raw/gh-pages/mi...[960rowsx22columns]