Package: emcstream
EmCStreamEmCStreamis a Python package designed for online embedding and clustering of evolving data streams.FeaturesImplements the EmCStream clustering algorithm, an adaptive clustering technique for evolving data streams.Uses UMAP (Uniform Manifold Approximation and Projection) for dimensionality reduction to maintain a low computational cost.Capable of detecting concept drifts in data streams and adjusting parameters accordingly, which makes it well-suited for non-stationary environments.Requires minimal parameter tuning, making it user-friendly for non-experts.DependenciesThis package depends on several well-known Python libraries for scientific computing and machine learning, including:numpyscikit-learnumap-learnInstallationBefore you can useEmCStream, you'll need to install it. Here's a step-by-step guide on how to do that:PrerequisitesEnsure you have the following installed on your system:Python 3.6 or laterpip (Python package installer)Installing EmCStreamOpen your terminalInstall the package using pip:pipinstallEmCStreamUsageThe following is a basic example of how to use the EmCStream package:fromEmCStreamimportEmCStream# specify paths to the dataset and labels filedataset_file_path="path_to_your_dataset"labels_file_path="path_to_your_labels"# specify horizon and number of clustershorizon=100k=3# create an EmCStream objectemcstream=EmCStream(dataset_file_path,labels_file_path,horizon,k)# run the clustering algorithmemcstream.run()# Datasets## Artificial DatasetsAllartificialstreamdatasetsusedinthisstudyarecreatedby*DSD\_RandomRBFGeneratorEvents*functionof**streamMOA**packageof**R**.50,000datainstancesarereceivedfromthestreamsandsavedtofiles.Thealgorithmsarefedbythedatainstancesonebyone,simulatingareal-timedatastream.Inthisstudy,wefocusonhighdimensionaldata,hencewehavecreatedhighdimensional,artificialstreamdatasets.Oneofthedatastreamsisstationary,whichmeansitiswithoutconceptdrift.Otherdatastreamsevolvewithdifferentspeeds,whichmeanstheyincludeconceptdrift.Eightstreamsarecreatedwithoutnoiseandthreestreamsarecreatedwithdifferentlevelsofnoise.Wehavespecifiedpropertiesofthestreamsinsuchawaythatitispossibletoobservetheeffectsofchangein*k*,dimensions,speedoftheconceptdriftandnoiselevel.Detailsofthedatastreamsareasfollows,where$k$indicates*numberofclusters*.|Streams|length|k|d|driftspeed*|noiselevel||-------------|-------------|-------|-------|-------------------------|---------------||Stream1|50.000|10|50|100|0||Stream2|50.000|10|100|100|0||Stream3|50.000|10|10|100|0||Stream4|50.000|20|50|100|0||Stream5|50.000|4|50|100|0||Stream6|50.000|10|50|40(highspeed)|0||Stream7|50.000|10|50|400(lowspeed)|0||Stream8|50.000|10|50|99999(nodrift)|0||Stream9|50.000|10|50|100|0.05||Stream10|50.000|10|50|100|0.10||Stream11|50.000|10|50|100|0.20|Wehavealsocreated5differentartificialdatastreamswithchangingnumberofclusters.Detailsofthesestreamsareasfollows.|Streams|length|k|d|driftspeed*|noiselevel||--------------|-------------|--------|-----|----------------|---------------||Changingk1|50.000|10+-2|10|100|0||Changingk2|50.000|10+-2|20|100|0||Changingk3|50.000|10+-2|50|100|0||Changingk4|50.000|20+-4|10|100|0||Changingk5|50.000|4+-1|10|100|0|\*:Kernelsmoveapredefineddistanceof0.01every*speed*points.__Note__:\*_X.txtfilesincludeonlydata,\*_labels.txtfilesincludeonlylabels,\*.csvfilesincludeboththedataandthelabels(lastcolumnicludesthelabels.)## Real Datasets### Meteorological DataWehavecomposedthreerealmeteorologicalstreamdatasetsusingweatherdatafromhttps://www.renewables.ninja/.(TheirtwodatasourcesareNASAMERRAreanalysisandCM-SAF's SARAH dataset). One of the datasets includes meteorological data of two cities from Turkey, other one includes two cities from Europe and the last dataset includes two cities from US. For each dataset, we chose two cities that have different climate characteristics to create separable datasets. All datasets consist of hourly measurements of five years, from Jan 1<sup>st</sup>, 2015, till Dec 31<sup>st</sup>, 2019. There exist 43,825 measurements for each city. Each dataset consists of two cities, therefore each dataset includes 87,650 instances. Every instance includes six features, which are temperature, precipitation, snowfall, snow mass, air density and cloud cover. Temperature and air density are two features that are always meaningful and distinguishing. However, other features are not always distinguishing. For example, precipitation is not a distinguishing parameter when it is not raining and snowfall or snow mass are not meaningful when it is not snowing. For this reason, we have defined an extra weight to temperature and air density. In order to apply this weight, we have doubled these two features, after normalizing the whole data.Forcreatingadataset,wehavemergedtheinstancesoftwocitiesaccordingtodateandtimeofthemeasurement.Bythisway,thedatasetsarerealstreamdatasets.Moreover,theseareevolvingdatastreamsbynature(theyincludeconceptdrift)becauseweatherdatachangesbothinadayof24hours,fromdaytimetonight,andinayear,fromseasontoseason.Usingsuchdatastreams,itispossibletofocuseitheronconceptdriftthatoccurseveryday,oronconceptdriftthatoccursfromseasontoseason.__Note__:\*_X.txtfilesincludeonlydata,\*_labels.txtfilesincludeonlylabels,\*.csvfilesincludeboththedataandthelabels(lastcolumnicludesthelabels.)### Keystroke DataWehaveusedthreesubsetsofthelargerCMUdatasetwhichiscreatedbytypingcharacteristicsof51users.Theparticipantstypethepassword".tie5Roanl"andthe*Enter*key400times,capturedineightdifferentsessionsindifferentdays,50timesoneachsession.Theoriginalkeystrokedatasetcontains20,400instancesandeachinstanceconsistsof31features.Keystrokedatasetincrementallyevolveduetotheparticipants' practice.WehavecreatedthreesubsetsoftheKeystrokedataset,gettingallfeaturesofsomeparticipants.Wedidnotchange,eliminateorconvertanyofthefeatures.Wehavespecified*k(numberofclusters)*astwo,threeandfour.Becauseeachparticipanthas400records,thesubsetsthatwehaveusedhave800,1,200and1,600instancesrespectively.__Note__:\*_X.txtfilesincludeonlydata,\*_labels.txtfilesincludeonlylabels,\*.csvfilesincludeboththedataandthelabels(lastcolumnicludesthelabels.)Followindtablepresentsasummaryofthecharacteristicsoftherealdatasets:|Stream|$k$|dimensions|instances|evolving||--------|-----|------------|-----------|----------||Weather-Turkey|2|6|87,650|yes||Weather-Europe|2|6|87,650|yes||Weather-US|2|6|87,650|yes||Keystroke-2|2|31|800|yes||Keystroke-3|3|31|1,200|yes||Keystroke-4|4|31|1,600|yes|