Package: rscylladb
Bulk Installer ScyllaDB/CassandraInsert over 20Million records within minutes.Special thanks 🤝 to @Reeya PatelInstall usingpipinstallrscyllaUsagefromrscylla.cqlimportCql# hostslist = ['127.0.0.1']# port = 9042# table_name = keyspace.table_name [must required]# username = cassandra# password  = cassandraobj=Cql(hostslist,port,table_name,username,password)# default chunks = 10,000# default workers = 4obj.insert(filename,chuncks,workers)# filename or filepathExample:fromrscylla.cqlimportCqlobj=Cql(["localhost"],9042,"cassandra","cassandra")obj.insert("data.csv")# optional : chuncks and workers# way 2obj.insert("data.csv",chunks=1000,workers=2)ResultTested on 4GB RAM with i3 processor.RowsDuration10,0000.03s3,00,0002m2,00,00,00066mNote: Depends on you hardware capability!Work flowImport packagesfromcassandra.clusterimportClusterfromcassandra.authimportPlainTextAuthProvider# authenticate userimportpandasaspdimportnumpyasnpimportthreadingimporttimeclass Cqlconstructor accept:list of hosts : list of stringport : integerusername : stringpassword : stringclassCql:def__init__(self,hosts:list,port:int,table_name:str,user:str,password:str)->None:In the constructor_ _ init _ _intialize cluster and authenticator for connect ScyllaDb server.self.start_time=time.time()# for completion the time show# if table name is not formated as keyspace.table_name then raise exceptioniflen(table_name.split("."))!=2:raiseException("Table name must be! keyspace.table_name")# table_name.split('.') returns [keyspace,table_name]# keyspaceself.keyspace_name=table_name.split(".")[0]# table nmeself.table_name=table_name.split(".")[1]# set authenticator with username & passwordauth=PlainTextAuthProvider(username=user,password=password)# set cluster, hosts,port and authself.cluster=Cluster(contact_points=hosts,port=port,auth_provider=auth)# set keyspaceself.session=self.cluster.connect(self.keyspace_name)insertmethod accept:For this method chunks and workers are optional# chunks default = 10000# workers default = 4definsert(self,file_name:str,chuncks=10000,workers=4)->None:Get inputed table's all columns and data types to avoid data type conflicts.get table's columns & data types fromsystem_schemakeyspace.table_schema=self.session.execute("SELECT column_name,type FROM system_schema.columns WHERE keyspace_name='%s' AND table_name='%s'"%(self.keyspace_name,self.table_name))Dictinary which contains cql data types & with python data types.cql data type as key & python data type as valueself.cql_to_python_type={'NULL':None,'boolean':bool,'float':float,'double':float,'int':int,'smallint':int,'tinyint':int,'counter':int,'varint':int,'bigint':int,'decimal':float,'ascii':str,'varchar':str,'text':str,'blob':str,'date':str,'timestamp':str,'time':str,'list':list,'set':set,'map':dict,'timeuuid,':None,'uuid':None}Createself.cql_typesdictionary forpandans Data Frame. Store all columns names from table_schema (exectracted fromschema_keyspace[inputed table's cols])self.cql_types={}self.cql_columns=[]# create dictonary for data frame: column_name:python_datatypeforrowintable_schema:# iterate extracted column_names & data types# row contains column_name & type# create key as column_name & value as python data type get from cql_to_python_type dictionaryself.cql_types[row.column_name]=self.cql_to_python_type[row.type]# add column name into listself.cql_columns.append(row.column_name)Example:# if student table contnains two rows:
    1. id : bigint
    2. name : text

    # which converted as
    cql_types = {
        "id":int,
        "name":str
    }Create commana seprated string for insert statementparam_keys=",".join(self.cql_columns)param_values=",".join(['%('+k+')s'forkinself.cql_columns])self.insert__statement="INSERT INTO "+self.table_name+" ("+param_keys+") VALUES ("+param_values+")"Example:if cql_columns = ['id','name']
then
param_keys = 'id,name'
# for bind value as dictionary (json object)
param_values = '%(id)s,%(name)s'
# param_values direct bind py execute method.Read file based on their extensionpandas read_csv method accept data frame & chunksize which slice huge file into chunks.supported files are: .csv, .json, .xls, .xlsx# supported filesfile_ext=file_name.split(".")[-1]# get extensioniffile_ext=="csv":# if csv then read csv with chuncksdata_frames=pd.read_csv(file_name,chunksize=chuncks)eliffile_ext=="json":data_frames=pd.read_json(file_name,chunksize=chuncks,lines=True)eliffile_extin["xls","xlsx"]:data_frames=pd.read_excel(file_name,chunksize=chuncks)else:raiseException("%sis not supported!"%(file_name))Example:data__frames has Pandas TextFileReader objectCreate empty thread listthreads=list(range(workers))# e.g workers = 4# threads = [0,1,2,3]Start while loop for read data_frames:df = next(data_frames) fetch data frame from TextFileReader, Every loop fetch new frameFirst for loop assigin thread object into threads[i] position & start threadIn the thread class we passexecute_insertmethod and parameters as args = (dataframe,)Second for loop wait for threads[i] complete the execution.whileTrue:try:# start threadforiinrange(workers):# fetch next dataframedf=next(data_frames)threads[i]=threading.Thread(target=self.execute_insert,args=(df,))threads[i].start()# wait until complete taskforiinrange(workers):threads[i].join()# on execption breakexceptExceptionase:print(e)breakCreated methodexecute_insert.Accept parameter DataFramedefexecute_insert(self,df:pd.DataFrame)->None:In the method:Make lower case column names. (cql have all lower case columns names)handle null type for numeric values:df.select_dtypes(include=np.number).columnsget column name which have data type float,intdf.select_dtypes(include=np.number).fillna(0)select columns & fill 0 if null presentdf.select_dtypes(exclude=np.number).fillna('')select non numeric cols and fill empty string.Iterate df (dataframe) withsession.execute_asyncmethod with insert statement & row(dict|json object).Methodsession.execute_asyncreturn future object so result() is required for completed execute.Addsession.execute_asyncintofutures listSecond loop iterate each future object & call result() method. By using this loop wait for allsession.execute_asyncmethod to complete execution.NOTE: In dataframe Pandas.NaN set by default where null value is located and cql raise exception so we have to fill 0 & empty string# make lower columns name for cqldf.columns=map(str.lower,df.columns)# handle null for numeric type default 0 adddf[df.select_dtypes(include=np.number).columns]=df.select_dtypes(include=np.number).fillna(0)# handle null for object and string typesdf[df.select_dtypes(exclude=np.number).columns]=df.select_dtypes(exclude=np.number).fillna('')# empty listfutures=[]# excute async & store future objectforrowindf.to_dict(orient='records'):# iterate each recordsfutures.append(self.session.execute_async(self.insert__statement,row))# wait for complete asyncforfutureinfutures:future.result()# wait until insert queriesAuthors :🙋‍♀️Reeya Patel🙋‍♂️Meet Rathod