Package: seeq-clustering
ContentsUser GuideInstallationseeq-clusteringis an add-on for Seeq Workbench, which allows for density based clustering of n-dimensional data. Clustering can be supervised (visual) or unsupervised (density based).User Guideseeq-clustering User GuideSeeq clustering determines cluster structure in data (either by finding regions of high density, or by manual definition) and returns each cluster as a Seeq condition. This can be useful for determiningregularly visitedregions in parameter space for a set of signals. Why this is important is best understood by analogy. If you want to know how your car's gas mileage is performing over time, a naive way to monitor it is by recording, at the end of every month, your total number of miles driven and total gas used that month. This will give you a measure of how your gas mileage performs over timebut is it a good measure of how your car is performing?Gas mileage depends on a variety of factors - outdoor air temperature, number of people in the car, speedetc- so a better measure of how your car is performing is by setting some set of parameters (e.g.70 degrees, driving between 25 and 30mph with only 1 person in the car) and calculating your gas mileageonlywhen your car is operating in that mode. By examing thisgas mileage during known conditionsover time, you have a much better measure of how your car is performing.This is precisely the aim of the clustering addon. In a real process it is more complicated than a car, and it is near impossible to know what set of conditions you should use to specify your monitoring periods (e.g.the 70 degrees, 1 person, from the car example). Clustering helps you with this. It gives you candidate parameter sets - regions in your data where spends a lot of time - so that you can compare how your process is performingin a known, and consistent, parameter space.When you open the tool, you will be met with two options:Visual (Supervised)- allows you to manually define a cluster boundary in 2-dimensional dataDensity (Unsupervised)- density based clustering to find clusters in n-dimensional dataYou may have to refresh the worksheetonce the addon tool says "SUCCESS". If only the conditions are displayed and not the original signals, please simply refresh the page.Visual (Supervised)Visual clustering is only applicable to 2-dimensional data. You simply define a region manually, and seeq-clustering will look for datapoints in that region. See an explanation forhow it worksDensity (Unsupervised)Seeexplanation of density based parametersfor detailed explanations of each input option for cluster definition.Whichever mode (Visual or Density) you choose, the form of the clusters returned in Seeq will look similar. Happy clustering!Explanation of Density Based ParametersMinimum Cluster Size (MCS)- the minimum number of datapoints which much be including in a cluster for it to be classified as a viable cluster. A larger number will produce fewer clusters, each of which then contains more data. SeeexamplesThere are two ways to specify MCS. Either by specifying a value (e.g."I want my clusters to contain no fewer that 200 points") or by a percent of the total number of data points (e.g."I want my clusters to contain no less that 5% of the data").notepercent of datapoints is calculated based on Display Range.You may (indeed, likely) not know what the total number of datapoints is in your data, so it is recommended to supply both aPercent of data (%)and aMin Cluster Points.If bothPercent of data (%)andMin Cluster Pointsare supplied, the default behavior will be to use thelargerof the two values for the MCS,i.e..You can override the behavior by checking theIgnore Percent of data?checkbox. Doing so will forcemcs = Min Cluster PointsDensity Based ExamplesTo demonstrate how min cluster size impacts the results on the clustering. We will show here two cases (both withIgnore Percent of data?checked)Min Cluster Points = 200Min Cluster Points = 100Errors:ValueError('unable to determine any cluster structure, try reducing mcs, or data size')No cluster structure was observed. The most likely problem is that the min cluster size (MCS) specified is too large or the dataset is too small. Try reducing MCS or increasing the dataset size (i.e.Display Range) slightly.How it works (Visual)When you define a region manually in visual based clustering, the algorithm defines a cluster region as follows:Selects the data in the Display Range within that regionFinds the center of the selected dataChooses random directions from the center, traverses along that direction until some threshold (usually 90%) of the selected data is "behind" (along that direction), places a boundary point there.It also uses the directionality of the walk to specify a "cone" of points which it classifies as "along the walk direction"Does this for many random walk directions to specify a set of contour points, like that shown here:This allows for a new test point (say, during calculation of Seeq condition) to be compared against the contour points to determine if that new test point is in the cluster or not. Owing to the nature of the random walk, this method is probabilistic and will not capture every member point 100% of the time. This membership definition is only used for visual clustering.Pseudo code for deterimining cluster membership:isinCluster(test_point, contour_points)
    Find the direction to the test_point from origin of cluster, r'
    Find the closest direction in the set of contour_points, r
      and corresponding length L
     
    calculate the length along r' to the test_point, L'
   
    if L'<L return true, else return falseInstallationThe backend ofseeq-clusteringrequiresPython 3.7or later.seeq-clusteringis compatible with Seeq >=R53.DependenciesYou will need to install theseeqmodule with the appropriate version that
matches your Seeq server. For more information on theseeqmodule seeseeq at pypiUser Installation Requirements (Seeq Data Lab)If you want to installseeq-clusteringas a Seeq Add-on Tool, you will need:Seeq Data Lab (>= R50.5.0, >=R51.1.0, or >=R52.1.0)seeqmodule whose version matches the Seeq server versionAccess (and permissions) to machine running Seeq serverKnowledge or whereexternal calculationscripts are located on that machine (seeUser Installationbelow)Enable Add-on Tools in the Seeq serverUser InstallationThese installation instructions are forSeeq >= 54, for R53 installation, seehere.
The latest source code of the project can be foundhere. The code is published as a courtesy to the user, and it does not imply any obligation for support from the publisher. After ensuring thatAdd-on tools are enabledin the Seeq server, follow the outlined steps below exactly.Create anewSeeq Data Lab (SDL) project and open theTerminalwindowInstall clusteringpipinstallseeq-clusteringor manually download the last.whlfile fromGitHub. Upload this to file to your SDL project, and install it:pipinstall<whl_file_name>Install the add-on toolpython-mseeq.addons.clusteringand follow the instructions when prompted. ("Username or Access Key" is what you use to log in to Seeq. "Password" is your password for logging into Seeq.)There are additionalOptionsfor the addon installation. These include--usersand--groups. These can be used to change permissions for the addon tool. For example to give permission to usersmeandyouone would install the addon with as:python-mseeq.addons.clustering--usersmeyouManually downloadthe two files (Basic.pyandconfig_ext_calc.py) from theexternal-calc folder.Ignorethe filesClustering.pyand_Clustering_config.py. Those are used for earlier Seeq version installations.Navigate to the external calculation folder on the machine where Seeq server is running, (typically'C:/Seeq/plugins/external-calculation/python/user/'or similar) andcreate a new foldercalledClusteringCalc.Move the two downloaded files (Basic.pyandconfig_ext_calc.py) into your newly created/ClusteringCalc/folder.In command line on the maching running Seeq (notseeq data lab terminal), navigate to the location of the files you just added (using the example from above):cd C:/Seeq/plugins/external-calculation/python/user/ClusteringCalcConfigure the location (on machine running Seeq Server) where clustering models will be stored. To do this, runpython config_ext_calc.py clusteringModelsPathIf you are unable to runpython config_ext_calc.py(e.g.if you do not have python installed on the Seeq server host machine), seemanual instructionsThe default is to store the models in the same location as dir as Clustering.py,i.e.D:/Seeq/plugins/external-calculation/python/user/in this example. If you wish to store your models elsewhere, and you have the required permissions Assuming you have permissions to access the path, this can be done by runningpython config_ext_calc.py clusteringModelsPath <yourpathhere>Follow the instructions in external-calc readme (typically located~/D:/ProgramData/Seeq/data/plugins/external-calculation/python/readme.html) to installhdbscan. Here is an exceprt from the readme, explaining how to do this:Installation of additional libraries can be done by executing the following steps:Stop Seeq if already startedYou may stop seeq by using the Seeq Workbench.Install the new Python module(s)Go to the place where Seeq Server is installed (usually C:\Program Files\Seeq Server) and runseeqprompt.batThis will open a new cmd window and will setup the Python environment for the next commands.To install your own Python libraries, run in this window (seeqprompt window) the following command:python -m pip install -U hdbscan -t plugins\lib\python3
### if you need to install additional packages, e.g. seeq, replace hdbscan with <packagename>where hdbscan is the name of the module we wish to install.Check and repair permissionsIf you run seeq as a service you will need to go to c:\ProgramData\Seeq\data\plugins\lib, select python3 folder, press right mouse, go to Properties , select Security tab, press Advanced, mark checkbox "Replace all child object permissions ..." press Apply, then Yes and then OK.Start SeeqOnce you've finished these steps the newly installed module may be used in your external-calculation Python scripts.If you run into an error in installation ofhdbscanseenoteToconfirmthat the external calculation has been installed correctly, in Seeq Workbench ensure that create a new formula and search forcluster. You should see the following:If you run into errors,please open an issueR53 InstallationThese are the installation instructions forR53.The latest source code of the project can be foundhere. The code is published as a
courtesy to the user, and it does not imply any obligation for support from the publisher. After ensuring thatAdd-on tools are enabledin the Seeq server, follow the outlined steps below exactly.Create anewSeeq Data Lab project and open theTerminalwindowInstall clusteringpipinstallseeq-clusteringor manually download the last.whlfile fromGitHub. Upload this to file to your SDL project, and install it:pipinstall<whl_file_name>Manually downloadthe two files (Clustering.pyand_Clustering_config.py) from theexternal-calc folderNavigate to the external calculation folder on the machine where Seeq server is running, (typically'C:/Seeq/plugins/external-calculation/python/user/'or similar). Move the two files you just downloaded (Clustering.pyand_Clustering_config.py) to this folder.In command line on the computer or server running Seeq (notseeq data lab terminal), navigate to the external calculation python folder (using the example from above):cd D:/Seeq/plugins/external-calculation/python/user/Configure the location (on machine running Seeq Server) where clustering models will be stored. Runpython _Clustering_config.py clusteringModelsPathThe default is to store the models in the same location as dir as Clustering.py,i.e.D:/Seeq/plugins/external-calculation/python/user/in this example. If you wish to store your models elsewhere, and you have the required permissions Assuming you have permissions to access the path, this can be done by runningpython _Clustering_config.py clusteringModelsPath <yourpathhere>If you are unable to runpython _Clustering_config.py(e.g.if you do not have python installed on the Seeq server host machine), seemanual instructionsFollow the instructions in external-calc readme (typically located~/D:/ProgramData/Seeq/data/plugins/external-calculation/python/readme.html) to installhdbscan. Here is an exceprt from the readme, explaining how to do this:Installation of additional libraries can be done by executing the following steps:Stop Seeq if already startedYou may stop seeq by using the Seeq Workbench.Install the new Python module(s)Go to the place where Seeq Server is installed (usually C:\Program Files\Seeq Server) and runseeqprompt.batThis will open a new cmd window and will setup the Python environment for the next commands.To install your own Python libraries, run in this window (seeqprompt window) the following command:python -m pip install hdbscan -t plugins\lib\python3
### if you need to install additional packages, e.g. seeq, replace hdbscan with <packagename>where hdbscan is the name of the module we wish to install.Check and repair permissionsIf you run seeq as a service you will need to go to c:\ProgramData\Seeq\data\plugins\lib, select python3 folder, press right mouse, go to Properties , select Security tab, press Advanced, mark checkbox "Replace all child object permissions ..." press Apply, then Yes and then OK.Start SeeqOnce you've finished these steps the newly installed module may be used in your external-calculation Python scripts.If you run into an error in installation ofhdbscanseenoteIn any Seeq workbook, retrieve the key of the newly created Clustering.py external calc call. Wait a few moments for it to update, you should see the external-calc script show up (your key will be calledClustering.py:NUMERIC:<your_unique_checksum>):Update your app to point to your unique checksum. Runpython -m seeq.addons.clustering.config --script_name <yourkeyhere>where<yourkeyhere>is that which you retrieved previously (from a Seeq workbook),e.g., the complete key:Clustering.py:NUMERIC:q2tYWyXR+cw7Install the add-on toolpython-mseeq.addons.clusteringand follow the instructions when prompted.There are additionalOptionsfor the addon installation. These include--usersand--groups. These can be used to change permissions for the addon tool. For example to give permission to usersmeandyouone would install the addon with as:python-mseeq.addons.clustering--usersmeyouTroubleshooting InstallManual external-calc Clustering ConfigTo manually update the cluster model path, open eitherBasic.py(>R54)Clustering.py(R53) in a text editor. The top two lines should be (or something similar):#DO NOT CHANGE THE FOLLOWING LINE OR THIS LINE
wkdir = ''Manually enter the path where you wish to save the clustering models by updating the wkdir variable (yes, this means ignoring the warning in the first line and YES change line 2. To replicate default behavior, updatedwkdirto be the sameabsolutepath as the directory which holdsBasic.py, e.g.:wkdir = 'D:/Seeq/plugins/external-calculation/python/user/ClusteringCalc/'Ensure that you use/in favor of\.Errors in hdbscan ext-calc installIf you encounter the errorerror: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
  ----------------------------------------
  ERROR: Failed building wheel for hdbscan
Failed to build hdbscan
ERROR: Could not build wheels for hdbscan which use PEP 517 and cannot be installed directlysimply follow the link provided in the error (https://visualstudio.microsoft.com/visual-cpp-build-tools/) and download and install Microsoft C++ Build Tools.NOTEit is highly recommended that you ensure Seeq server is stopped before installing Microsoft C++ Build Tools. Follow the download instructions and install the defaults for Microsoft C++ Build Tools:You will likely then have to close your command prompt, and rerunseeqprompt.batbefore attempting to install hdbscan again (seesteps)TroubleshootingErrorSPyRuntimeError: Error pushing "[39575A91-37A5-4461-AC8C-7821B54B4237] {Condition} Cluster 0 01/20/2022, 06:13:12": Failed to write item in batch request: Invalid script reference. Expected: fileName.{py|xlsx|xlam|xlsm}[:functionName]:resultDataType:checkSum at 'externalCalculation', line=1, column=1This occurs from a change to how external calculations function that occurred after Seeq R54. This can be overridden by (i.e., you can force the R54 usage style):python-mseeq.addons.clustering._external_calc_override--overrideTrueIf, for some reason you wish to undo this change:python-mseeq.addons.clustering._external_calc_override--overrideFalseDevelopmentWe welcome new contributors of all experience levels. TheDevelopment Guidehas detailed information about
contributing code, documentation, tests, etc.Important linksOfficial source code repo:https://github.com/seeq12/seeq-clusteringIssue tracker:https://github.com/seeq12/seeq-clustering/issuesSource codeYou can get started by cloning the repository with the command:gitclonegit@github.com:seeq12/seeq-clustering.gitSupportCode related issues (e.g. bugs, feature requests) can be created in theissue trackerMaintainer: Eric ParsonnetCitationPlease cite this work as:seeq-clustering
SeeqCorporation,2021https://github.com/seeq12/seeq-clustering