Package: data-ecosystem-python
PADE-Python Project DocumentationPoint of contact:John BowyerOrganizational unit: OCIORelated projects: EDCRelated investments:  Pending Public ReleaseGovernance status: Pending Public ReleaseProgram official:Erik KnudsenGetting StartedSet up Local development environment for Python - with remote sparkCheck if Python is installedRun in bash or powershellCheck Python Version```sh
 python3 --version
 ```
 
 or
 
 ```sh
 python --version
 ```Ensure it is python 3.9.9Install Python 3.9Run Install Python on Ubuntu or WSL (Primary)```sh
    sudo apt update
    sudo apt install python3.9
    ```Update Path```sh
    nano ~/.bashrc
    export PATH="/usr/bin/python3.9:$PATH"
    source ~/.bashrc
    ```Run Install Python on windowsRun ApplicationDownload: https://www.python.org/ftp/python/3.9.9/python-3.9.9-embed-amd64.zip
    Install as a non global userInstall PipRun Install Pip on Ubuntu or WSL (Primary)```sh
    sudo apt update
    sudo apt-get install python3-pip
    ```Run Install Pip on Windows```sh
    py -m ensurepip --upgrade
    ```Install Virtual EnvironmentRun Install Virtual Environment on Ubuntu or WSL (Primary)SeeVirtual Environment Set Reference ArticleIf first virtual environment on desktopOpen Terminal in home directory```sh
 cd $HOME
 ```Create Directory for virtual environmentmkdir.virtualenvEnsure Pip is installedsudoaptinstallpython3-pipValidate pip is installedpip3--versionInstall virtualenv via pip3.pip3installvirtualenvValidate virtualenv locationwhichvirtualenvInstall 3.9 venvsudoaptinstallpython3.9-venvInstall Virtual Environments Wrapperpipinstallvirtualenvwrapper
pip3installvirtualenvwrapperUpdate pathexportPATH="$HOME/.local/bin:$PATH"exportPATH="/usr/bin:$PATH"echoWORKON_HOME="~/.virtualenvs"|sudotee-a$HOME/.bashrcechoVIRTUALENVWRAPPER_VIRTUALENV="$HOME/.local/bin/virtualenv"|sudotee-a$HOME/.bashrcecho"VIRTUALENVWRAPPER_PYTHON=$(whichpython3)"|sudotee-a$HOME/.bashrcecho"source$HOME/.local/bin/virtualenvwrapper.sh"|sudotee-a$HOME/.bashrcsource$HOME/.bashrcIf second or later virtual environment on desktopMake Virtual EnvsmkvirtualenvOCIO_PADE_DEV
mkvirtualenvEZDX_FOODNET_DEV
mkvirtualenvOCIO_PADE_DEV
mkvirtualenvOD_DATALAKE_DEV
mkvirtualenvOD_NHANES_DEV
mkvirtualenvNDSP_PERTUSSIS_DEV
mkvirtualenvNDSP_PERTUSSIS_QA
mkvirtualenvNDSP_PERTUSSIS_ONBOARD
mkvirtualenvNDSP_PERTUSSIS_PRODIf existing environment configured - deactivatedeactivate
sudopipuninstallnodeenv
pipinstallnodeenv
workon{virtualenvname}nodeenv-pExampledeactivatecd$HOME# cleanup any existing nodesudorm-rfnode-v18.12.1-linux-x64.tar.xz
sudorm-rf/usr/local/bin/npm/usr/local/share/man/man1/node*~/.npm
sudorm-rf/usr/local/lib/node*
sudorm-rf/usr/local/bin/node*
sudorm-rf/usr/local/include/node*
sudoapt-getpurgenodejsnpm
sudoaptautoremove
sudopipuninstallnodeenvActivate new virtual envdeactivatecd$HOME# cleanup any existing nodesudorm-rfnode-v18.12.1-linux-x64.tar.xz# sudo rm -rf /usr/local/bin/npm /usr/local/share/man/man1/node* ~/.npm# sudo rm -rf /usr/local/lib/node*# sudo rm -rf /usr/local/bin/node*# sudo rm -rf /usr/local/include/node*sudoapt-getpurgenodejsnpm
sudoaptautoremove
sudopipuninstallnodeenv
workonVIRTUAL_ENV# EXAMPLE:workonOCIO_PADE_DEVcd$VIRTUAL_ENV# install nodewgethttps://nodejs.org/dist/v18.12.1/node-v18.12.1-linux-x64.tar.xz
tar-xfnode-v18.12.1-linux-x64.tar.xz
sudomvnode-v18.12.1-linux-x64/bin/*./bin/
sudomvnode-v18.12.1-linux-x64/lib/node_modules/./lib/node_modules/# Verify installation usingnode-v
npm-v
npminstallnpm@9.1.1
pipinstall--upgradepip
pipinstallnodeenvcd$VIRTUAL_ENV/lib/node_modules/npm
nodeenv-pcd$VIRTUAL_ENV/lib
npminstall@mermaid-js/mermaid-cli--registry=https://registry.npmjs.org
cdvirtualenvbinecho"export PATH='$PATH:$VIRTUAL_ENV/lib/node_modules/.bin'"|sudotee-aactivatesourceactivate# Test clientmmdc-hRemove local Virtual EnvironmentRun remove virtual environment on Ubuntu or WSL (Primary)Rundeactivate
rmvirtualenvOCIO_PADE_DEVFor globalsudorm-rfvenvFor localrm-rfOCIO_PADE_DEVRun Unit Test Coverage ReportRun Unit Test Coverage Report on Ubuntu or WSL (Primary)Run the following commandcddata_ecosystem_python
pytest--cov-reporthtmltests/Set up Local development environment for DockerInstall Docker without License on Ubuntu or WSL (Primary)Reference 1:Install Docker Engine without Docker DesktopReference 2:Install Docker for WSLOption 1: Convenience ScriptDocker provides a convenience script atDocker Scriptto install Docker into development environments non-interactively. The convenience script isn't recommended for production environments, but it's useful for creating a provisioning script tailored to your needs.Run commandcurl-fsSLhttps://get.docker.com-oget-docker.sh
sudoshget-docker.shNote: You will have to wait 20 secondsBuild and Run Imagecd.devcontainerlocal
sudodockerbuild-tOCIO_PADE_DEV.
sudodockerrun-it--rm--nameOCIO_PADE_DEVOCIO_PADE_DEVOption 2: Manual InstallCheck that you have WSL version 2wsl--set-default-version2Remove any old docker imagessudoaptremovedockerdocker-enginedocker.iocontainerdruncInstall/Upgrade dependenciessudoapt-getupdate
sudoapt-getinstall\ca-certificates\curl\gnupg\lsb-releaseAdd Docker's official GPG keysudomkdir-p/etc/apt/keyrings
curl-fsSLhttps://download.docker.com/linux/ubuntu/gpg|sudogpg--dearmor-o/etc/apt/keyrings/docker.gpgAdd Docker's stable repositoryecho"deb [arch=$(dpkg--print-architecture)signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu$(lsb_release-cs)stable"|sudotee/etc/apt/sources.list.d/docker.list>/dev/nullInstall Docker Engine and Docker Composesudoapt-getupdate
sudoapt-getinstalldocker-cedocker-ce-clicontainerd.iodocker-compose-pluginVerify Docker Installsudodockerrunhello-worldInstall Docker without Desktop License on WindowsDocker CLI is required for managing images. It is available for install atSet up Local development environment for EcPaasInstall OpenshiftThe OC Client is used to manage all resources in a project and can be downloaded fromInstallOC ClientThe OCP Web sh can also be used to perform most of the project administration tasks. Additional OC information is available atOpenShift ClientInstall OpenShift Postgres SetupConnection instructionOpenShift Client Postgres connectionSteps to follow:Login to ARO *RedHat Open ShiftOpenShift loginconnect to pod nccdphp-postgresql-test-26-xppw8RunocLogin to:OpenshiftSelect AAD to log in.Once logged in, Select "Copy Login command"top right menu. This opens another login screen,select AADclick "Open token"Copy login token to cmd oc screenRun login commandChange to correct project:ocproject"ddt-ops-center"Port-forward usingocport-forwardnccdphp-postgresql-test-32-btxvk5432(if the pod has changed, find the correct pod name using the commands below)Get list of pods using:ocgetpo
ocgetpo|greppost(ifyouareusinggitbashasyourshell)ocgetpo|findstr"post"(selectonlyPostgreSQLpods)While this is running connect via Azure Data Studio
Credentials sent seperatelyOpenShift Client Postgres connection Troubleshootingif stuck on forwardingclick on server name in azure data studioSet up Local development environment for Node Web ApplicationsInstall Node on WindowsRunNode 18.12.1 Installation MSIRun Update Node to latest versionnpminstall-gnpmInstall PoetryRun Install Poetry on Ubuntu or WSL (Primary)Reference:StackOverflow: Poetry with DockerInstall Poetrycd$VIRTUAL_ENVcurl-sSLhttps://install.python-poetry.org|python3-Copy poetry to local bincp$VIRTUAL_ENV/bin/poetry~/.local/binRun Install Poetry on Windows(Invoke-WebRequest-Urihttps://install.python-poetry.org-UseBasicParsing).Content|py-Activate Virtual EnvironmentRun activate virutal environmentworkonEZDX_FOODNET_DEV
workonOCIO_PADE_DEV
workonOD_DATALAKE_DEV
workonNDSP_PERTUSSIS_DEV
workonNDSP_PERTUSSIS_QA
workonNDSP_PERTUSSIS_ONBOARD
workonNDSP_PERTUSSIS_PRODUpdate Libraries with PoetryPrerequisites for Run Update Libraries on Ubuntu or WSLClone Repository DAVTOpen Repository in VS CodeSet Python environment to a virtual environment such OCIO_PADE_DEVOpen Terminal in WSLRun Update Libraries on Ubuntu or WSL (Primary)cd_templates/data_ecosystem_python
poetryupdateRun Update Libraries on Windowscd"C:\Users\jcbow\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer"poetryupdateVS Code Python SettingsGo to VS Code -> File -> Preferences -> Settings -> Workspace Settings -> Python ConfigurationClick Edit in settings.jsonSearch for each of the following settings to updateAdd this line to your settings: "python.linting.pep8Args": ["--max-line-length=120"],Add this line to your settings: "python.linting.pep8Args": ["--ignore=E402,F841,F401,E302,E305"],Add this line to your settings: "python.analysis.typeCheckingMode": "basic"VS Code Markdown SettingsFor the markdown all in one exensionSet markdown.extension.toc.levels to 2..6 to skip headerSet up Local Environment .NET Core for Web API DevDeveloper/Data Engineer Project Quick StartDecide on two part name of projectRoot Project: Example: ocioIndividual Project: Example: padeGet access to center SharePoint and set up clone initial SharePoint lists if this is the first DAVT project in the centercodes/value_setscolumnsdatasetsenvironmentsjobspipelinesprojectsreportsusersuser_rolesCreate GitHub repository in GitHub CDCEnt organization named {root_project}_{individual_project}: Example: ocio_padeClone GitHub repository to local machine and open in VSCodeCreate docs directory in root of project and copy standard filesconf.pydavt_t.docxdavt_t.pptxindex.rstlua_word.lualua_pdf.lualua_html_pptx.lualua_md.luaRequest top level a storage container in EDAV if this is the first EDAV project in the centerConfigure top level project folder in EDAV storage containerSet up top level foder: example:ocio_padeSet up dev and qa subfolders: example:ocio_pade/devSet up ingress, archive and config subfolders: example:ocio_pade/dev/ingressocio_pade/dev/archiveocio_pade/dev/configDeveloper/Data Engineer Troubleshoot Problems and ErrorsProblem: Java: Windows ignores JAVA_HOMESymptom:   Windows ignores JAVA_HOMEResolution:The Java installer will put a copy of java.exe (but no libraries) in the C:\Program Files (x86)\Common Files\Oracle\Java\javapath directory and add that directory to the beginning of the PATH variable.If you don"t use a full path, the copy of java.exe to run is found by using the PATH system variable. Since this directory doesn"t contain the DLLs of a particular Java runtime version, one is located one by looking at the registry.So, you either need to modify the registry, or replace the javapath entry with the version of Java you want in your PATH system (not user) variable.Recommend setting registryComputer\HKEY_CURRENT_USER\Environment\JAVA_HOME
    C:\apps\Java\jdk1.8.0_333Reference:Stack OverflowProblem: Node: Unable to remove existing node_modulesProblem: Unable to remove existing local node_modulesSolution:Run Commandcdpath_with_modules
find.-name'node_modules'-typed-prune-execrm-rf'{}'+Problem: Unable to remove all existing global node_modulesSolution:Run Commandnpmls-gp--depth=0|awk-F/'/node_modules/ && !/\/npm$/ {print $NF}'|xargsnpm-grmProblem: Python: Unable to remove existing python packagesProblem: Unable remove existing python packagesSolution:  Run from elevated command prompt to removes all local packages in your environmentpipfreeze|xargspipuninstall-yProblem: NPM Won't Run After UpgradeProblem: NPM Won't Run After Upgrade using a manual installReference:Stack OverflowSolution:If npm is no longer installed in /usr/bin/npm, then chances are good bash(1) has hashed the executable name. The hashing saves repeated searches of all directories in your PATH every time you execute common programs. Since programs almost never change directories, this is usually a great idea.  To fix this isue, remove the hash by running the following command:hash-r.Problem: WSL: Ubuntu: Unable to reset WSL root password in UbuntuSymptom:  Unable to reset WSL root password in UbuntuReference:AskUbuntuSolution:Open cmd.exeType:wsl-urootChange the password:passwdusernameType:exitType:wslConfirm the new password works:sudoechohi
spark.conf.set("spark.databricks.pyspark.enablePy4JSecurity","false")Error: Azure B2C: The subscription is not registered to use namespace 'Microsoft.AzureActiveDirectory'.Symptom: Receive error when creating an Azure B2C tenant: The subscription is not registered to use namespace 'Microsoft.AzureActiveDirectory'.  Seehttps://aka.ms/rps-not-foundfor how to register subscriptions.Reference:BlogResolution:Login to azure. Start up a shell and type the following command.az loginThis will start up a browser and you will need to log into Azure.Get subscription list.  If you have one subscription then the default will be selected. If not then you will need to check your subscriptions and then set the right one.  Using the account list command you can get a list of your subscriptions.az account listYou will get a list of subscriptions that you have access to.  You can then set the subscription you want to work on.Set the subscription to work on.az account set --subscription "ZFI4-PERSONAL-DEV"Register the namespace provideraz provider register --namespace Microsoft.AzureActiveDirectoryError: Azure: ClientSecret: Secret should be an Azure Active Directory application's client secretSymptom:  Receive error: secret should be an Azure Active Directory application's client secret when calling function: credential = ClientSecretCredential(tenant_id, client_id, client_secret)Resolution:- If running from local jupyter notebook ensure you have set the local python environment for your project: Example: OD_NHANES_DEV
    - Use the client secret of the service principal, not the client secret of the app registration.
    - Ensure that the client secret of the service principal is set in the system environment variable for your application: Example: DAVT_OD_NHANES_DEV_AZURE_CLIENT_SECRETError: Databricks: Spark: Cannot find catalog plugin class for catalog "spark_catalog": org.apache.spark.sql.delta.catalog.DeltaCatalogProblem: org.apache.spark.SparkException: Cannot find catalog plugin class for catalog "spark_catalog": org.apache.spark.sql.delta.catalog.DeltaCatalogResolution:From your virtual environment.  You can select using workon venv_nameRunpyspark--packagesio.delta:delta-core_2.12:1.0.0--conf"spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension"--conf"spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"orCopy io.delta:delta-core_2.12:1.0.0 JAR file to $SPARK_HOME/lib and restartThis should make error go away.Reference:Stack OverflowError: Databricks: Error in SQL statement: SecurityException: User does not have permission SELECT on any fileSymptom: Table creation fails in Databricks with security exception: Error in SQL statement: SecurityException: User does not have permission SELECT on any file.Reference:Databricks KBSolution:Run the following command:GRANTSELECTONANYFILETO`user1`WarningIt is important to understand the security implications of granting ANY FILE permissions on a filesystem. You should only grant ANY FILE to privileged users. Users with lower privileges on the cluster should never access data by referencing an actual storage location. Instead, they should access data from tables that are created by privileged users, thus ensuring that Table ACLS are enforced.In addition, if files in the Databricks root and data buckets are accessible by the cluster and users have MODIFY privileges, the admin should lock down the root.Error: Databricks: Spark: Illegal character in pathProblem: Spark illegal character in pathResolution:runspark-classorg.apache.spark.deploy.master.Masterthenspark-shell--masterspark://ip_and_port_from_step_aboveexamplespark-shell--masterspark://192.168.86.27:7077Reference:Stack OverflowError: Databricks: Spark: Cannot find catalog plugin classSymptom:  Receive error: Cannot find catalog plugin class for catalog "spark_catalog"Resolution:Copy delta-core_2.13-2.1.0 JAR file to $SPARK_HOME/lib and restart, this error goes away.Error: Databricks: Spark: WARN ProcfsMetricsGetterSymptom:  Receive errror:  Encountering "WARN ProcfsMetricsGetter: Exception when trying to compute pagesize" error when running SparkResolutionAdding PYTHONPATH environment variable with value as:%SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-<version>-src.zip;%PYTHONPATH%Reference:Stack OverflowError: Databricks: Spark: py4j: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelistedProblem: Error: py4j.security.Py4JSecurityException: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelisted when connecting from Databricks to ADSL from Datascience ClusterReference:Databricks Community Q and ASolution:Update function setup_spark_configuration in environment_metadata classspark.conf.set("spark.databricks.pyspark.enablePy4JSecurity","false")Error: Databricks: SQL: Table creation fails with security exceptionProblem: Table creation fails with security exceptionReference:Databricks KBSolution:GRANTSELECTONANYFILETO`user1`WarningIt is important to understand the security implications of granting ANY FILE permissions on a filesystem. You should only grant ANY FILE to privileged users. Users with lower privileges on the cluster should never access data by referencing an actual storage location. Instead, they should access data from tables that are created by privileged users, thus ensuring that Table ACLS are enforced.In addition, if files in the Databricks root and data buckets are accessible by the cluster and users have MODIFY privileges, the admin should lock down the root.Error: Docker: ERROR: failed to solve: error getting credentials - err: docker-credential-desktop.exe resolves to executable in current directory (./docker-credential-desktop.exe), out:Symptom: During Docker Build Receive ERROR: failed to solve: error getting credentials - err: docker-credential-desktop.exe resolves to executable in current directory (./docker-credential-desktop.exe), out:Reference:Stack OverflowSolution:Run command:code~/.docker/config.jsonChange credsStore to credStoreSaveError: Docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?Symptom: Receiving error: "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? " when trying to run a docker build commandReference:Stack OverflowSolution:Run command:sudodockerdError: Docker: During connect receive message: This error may indicate that the docker daemon is not runningSymptom:  Receive error: error during connect: This error may indicate that the docker daemon is not running.: when attempting to run docker imageReference:Stack OverflowDetails:The error is related to message: In the default daemon configuration on Windows, the docker client must be run elevated to connectResolution with Docker Desktop:Verify that Docker Desktop application is running. If not, launch it: that will run the docker daemon (just wait few minutes).If the error still persists, you can try to switch Docker daemon type, as explained below:Resolution with Powershell:Open Powershell as administratorLaunch command: & 'C:\Program Files\Docker\Docker\DockerCli.exe' -SwitchDaemonResolution with CMD:Open cmd as administratorLaunch command: "C:\Program Files\Docker\Docker\DockerCli.exe" -SwitchDaemonError: Git: remote: error: File x is 296.35 MB; this exceeds GitHub's file size limit of 100.00 MBSymptom: Receive error: git push origin dev-zfi4:dev-zfi4
remote: warning: File pade/pade_ocio_ingress/ehr/encounters.csv is 87.59 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: Trace: 2c8fbfed93d4f4c52468dde090cd8d14a1a311d13a6aa58ea35cb9cb8c5d2577
remote: error: Seehttp://git.io/iEPt8gfor more information.
remote: error: File pade/pade_ocio_ingress/ehr/observations.csv is 296.35 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage -https://git-lfs.github.com.
Tohttps://github.com/cdcent/data-ecosystem-services.git! [remote rejected] dev-zfi4 -> dev-zfi4 (pre-receive hook declined)Reference:Stack OverflowSolution:gitfilter-branch--tree-filter'rm -rf path/to/your/file'HEAD
gitpushError: Git: Support for password authentication was removed. Please use a personal access token insteadSymptom:  Receive error: Support for password authentication was removed. Please use a personal access token instead. when trying to push to a remote repositoryReference:Stack OverflowSolution:For Ubuntu:For Linux, you need to configure the local GIT client with a username and email address,gitconfig--globaluser.name"your_github_user_id"gitconfig--globaluser.email"your_github_email"gitconfig-lError: Git: Tag: Would clobber existing tagSymptom:  CICD or VS Code Tag give error "would clobber existing tag"Solution:Run Gitgitfetch--tags-fThen pull again.Reference:Stack OverflowError: Git: fatal: mmap failed: Invalid argumentSymptom: Receiving error: Git fatal: mmap failed: Invalid argumentReference:Stack overflowSolution:Log into OneDriveFinish synchError: Java: Gateway process exited before sending its port numberProblem:  "Java gateway process exited before sending its port number"Resolution:On All OS:Check that the following environment variable is set where 1 is the number of processorsexportPYSPARK_SUBMIT_ARGS="--master spark:192.168.86.27:7077"orexportPYSPARK_SUBMIT_ARGS="--master local[1] pyspark-shell"On Ubuntu: install openjdk-8-jdk packagesudoapt-getinstallopenjdk-8-jdk-headless-qqOn MacOs: run:java-versionVerify Resolution In Python:frompysparkimportSparkContextsc=SparkContext.getOrCreate()# check that it really works by running a job# example from http://spark.apache.org/docs/latest/rdd-programming-guide.html#parallelized-collectionsdata=range(10000)distData=sc.parallelize(data)distData.filter(lambdax:notx&1).take(10)ORReference:Stack OverflowError: Java: Maven: javax.net.ssl.SSLHandshakeException: PKIX path building failedSymptom:   Server access error at urlMaven URL(javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)Resolution:Run:From cli change dir to jre\binFirst of all you copy the URL that you are connecting to and paste it in your browser. Just paste the url in the address bar and press enter.
Example:Maven RepoNow click on the lock button on the left of the url to see Certificate (valid)Click on View Certificate to open up the following popupClick on the Certification Path tab, where you will see the certificate chainDouble click on the parent certificateCheck keystore (file found in C:\apps\jdk-8.0.342.07-hotspot\jre\lib\security directory)
a. cd C:\apps\jdk-8.0.342.07-hotspot\bin
b. keytool -list -keystore cacerts
c. Password is changeit
d. Download and save all certificates in chain from needed server.Remove "read-only" attribute on file ..\lib\security\cacerts),Copy repo1.maven.org.crt to "C:\apps\Java\jdk1.8.0_333\lib\security\repo1.maven.org.crt" and "C:\Users\zfi4\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer_templates\certs\repo1.maven.org.crt"Add certificateskeytool-import-aliasrepo1_maven_org-keystoreC:\apps\jdk-8.0.342.07-hotspot\jre\lib\security\cacerts-file"C:\Users\zfi4\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer\_templates\certs\repo1.maven.org.crt"keytool-import-aliasrepo_spark_packages_org-keystoreC:\apps\jdk-8.0.342.07-hotspot\jre\lib\security\cacerts-file"C:\Users\zfi4\OneDrive - CDC\DAVT Analytics\davt-dev-jcbowyer\_templates\certs\repos.spark-packages.org.crt"Run the keytool command again to verify that your private root certificate was addekeytool-list-keystoreC:\apps\Java\jdk1.8.0_333\lib\security\cacertsReference:Stack OverflowReference:Java SamplesError: Logic Apps: O365 User Name and Password Entry Window Disappears on SharePoint Connector SetupThe pop-up window for user name and password entry may close immediately when Cached Credentials expired.Try deleting your cached O365 credentials in Credentials Manager.If clearing O365 casched credentials does not work, try deleting certificates.If neither deleting you cached O365 credentials nor deleting ceritifates works, try deleting the identities folder that may still be cached:Close all office applicationsPress Win+R and type regedit to open Registry EditorFrom Registry Editor, browse to: KEY_CURRENT_USER\Software\Microsoft\Office\16.0\Common\Identity\IdentitiesDelete the identities folder.Open Office and sign in again to check if the same problem will occur.Error: Logic Apps: Save logic app failed: 'Execute_JavaScript_Code' of type 'JavaScriptCode'Symptom:  Receive Error:  Failed to save logic app logic-sps-list-export-all-environments. The workflow must be associated with an integration account to use the workflow run action 'Execute_JavaScript_Code' of type 'JavaScriptCode'.Reference:Stack OverflowResolution:  To run the inline code action you need an integration account. These have some different pricing tiers. You only need to create it and then on the Logic App Workflow settings associate it.Error: Mermaid: Expecting 'NEWLINE', 'SPACE', 'GRAPH', got 'ALPHA'Problem: Mermaid raises error Expecting 'NEWLINE', 'SPACE', 'GRAPH', got 'ALPHA' when it encounters bracket.Reference:GitHubThe issue seems to be, that mermaid.cli ships with an old version of mermaid.min.js which doesn't support these kinds of node types/attributes.Resolution:Explicitly install mermaid and then copy dist/mermaid.min.js into the mermaid.cli folder within node_modulesVersion is avaialble atunpkg.com-mermaidError: Node: NPM: PhantomJS not found on PATHSymptom:  Receiving error: "PhantomJS not found on PATH" when running npm installReference:StackOverflowSolution:Run commandsudonpmuninstallphantomjsRun commandsudonpminstallphantomjs-gIf fails global install try local installsudonpminstallphantomjs-sError: Python: Virtualenv - workon command not foundSymptom: workon command not foundReference:StackOverflowDetails:The workon command is not available if you have restarted the shell.If you want this to work with each shell, you'll need to add these to your ~/.bashrc fileSolution:RunexportPATH="$HOME/.local/bin:$PATH"echo"export WORKON_HOME=$HOME/.virtualenvs"|sudotee-a$HOME/.bashrcecho"VIRTUALENVWRAPPER_PYTHON='/usr/lib/python3.9'"|sudotee-a$HOME/.bashrcecho"source$HOME/.local/bin/virtualenvwrapper.sh"|sudotee-a$HOME/.bashrcsource$HOME/.bashrcError: Puppeteer: can't launch chromium, missing shared library libgbm.soSymptom: Receiving error: "Error: Failed to launch the browser process! /usr/bin/chromium: error while loading shared libraries: libgbm.so.1: cannot open shared object file: No such file or directory"Details:  Puppeteer is a Node library which provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.  It is not launching when running mermaid-cli.Reference:GitHubSolution:Add the lgm package to the Dockerfile/Setup processsudoapt-getupdate
sudoapt-getinstall-ylibgbm-devError: Python: Current Python version (3.8.10) is not allowed by the project (^3.9).Symptom: Error: Python: Current Python version (3.8.10) is not allowed by the project (^3.9). Current Python version (3.8.10) is not allowed by the project (^3.9). Please change python executable via the "env use" command.Reference:Stack OverflowDetails:  Please change python executable via the "env use" command.Solution:You can get the path to your Python version by runningwhichpython3.9To update your environment to latest path runpoetryenvuse/usr/bin/python3.9Error: React: NodeJS: Ubuntu: Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/node-sass/build'Symptom: Receiving Error when running npm link command:  Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/node-sass/build'Reference:Stack OverflowDetails: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/node-sass/build'Solution:Change the ownership of folder node_modules, because you use sudo npm install -g node-sass, so its ownership is set to root.sudochown-Rroot:$(whoami)/usr/local/lib/node_modules/
sudochmod-R775/usr/local/lib/node_modules/Error: Sphinx: WARNING: dot command 'dot' cannot be run (needed for graphviz output), check the graphviz_dot settingSymptom: Receiving error: "WARNING: dot command 'dot' cannot be run (needed for graphviz output), check the graphviz_dot setting" when building HTML Sphinx documentationReference:Stack OverflowReference:BlogDetails: The problem could be referred to an incorrect configuration of "dot" executable from GraphViz PATH.Solution:Adding the PATH using the export command, such as:exportPATH=$PATH:~/opt/binError: Ubuntu: InRelease is not valid yet (invalid for another 1d 21h 40min 43s). Updates for this repository will not be applied.Symptom:  Ubuntu: InRelease is not valid yet (invalid for another 1d 21h 40min 43s). Updates for this repository will not be applied. occurs when running sudo apt-get updateReference:StackOverflowDetails: Occurs when the system clock in Ubuntu is not CorrectSolution: Run the following command to fix the issue:sudodate-s"$(wget-qSO---max-redirect=0google.com2>&1|grepDate:|cut-d' '-f5-8)Z"sudoaptupdateError: Ubuntu: System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is downSymptom:  Receiving error: "System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down" when running : sudo systemctl daemon-reloadReference:Github GistSolution: Run the following command to fix the issue:sudoapt-getupdate&&sudoapt-getinstall-yqqdaemonizedbus-user-sessionfontconfig
sudodaemonize/usr/bin/unshare--fork--pid--mount-proc/lib/systemd/systemd--system-unit=basic.targetexecsudonsenter-t$(pidofsystemd)-asu-$LOGNAMEError: Ubuntu: APT: Release file forUbuntu Fileis not valid yetSymptom:  Receiving error: "Release file forUbuntu Fileis not valid yet" during apt-get updateReference:How2ShoutSolution:Check your date and time settings. If they are not correct, then you can fix them by running the following command:sudodpkg-reconfiguretzdataError: VS Code Ubuntu/Docker/Web Server: Failed to bind to address already in useSymptom:  Deploying a project to a local web server return error message: VS Code Ubuntu/Docker/Web Server: Failed to bind to address already in useReference:Stack OverflowSolution:Determine which ports are blockedlsof-i:<portnumber>example:lsof-i:5001Kill the process that is blocking the portkill-9<processnumber>example:kill-91600Error: Ubuntu: APT: Permission Denied when calling apt-get update from WSLSymptom:  Permission Denied when calling apt-get updatee from WSLDetails:Run wsl command from ubuntuReceive ErrorReading package lists... Done
E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
E: Unable to lock directory /var/lib/apt/lists/Reference:AskUbuntuThis line says you are not authorized to install/update. You need to be root to do it....Resolution with WSL:Add sudo before apt-get or you can do sudo -s to be root.sudoapt-getupdateorRun the following command to promote the current terminal to run as root until the terminal is closed.sudo-s
apt-getupdateError: Ubuntu: ImportError: No module named pip ' right after installing pipSymptom: Receive error: "ImportError: No module named pip" right after installing pipReference:Stack OverflowDetails: /home/jcbow/.virtualenvs/OCIO_PADE_DEV/bin/python: No module named pipResolution:First, ensure that python is included in the PATH variable, then run
Then run the followingpython-mensurepipError: Ubuntu: Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknownProblem: When browsinghttps://127.0.0.1:5000receive error: SSL: Handshake failed. while not logged into ZScalerSolution:Here are the steps you can follow to fix this error:Check if the UPower daemon is installed on your system by running the following command in the terminal:systemctlstatusupower.serviceIf the command output shows that the UPower service is not running or is not installed, you can install it using the package manager for your system (e.g. apt-get for Ubuntu/Debian, dnf for Fedora, pacman for Arch Linux, etc.).Install the UPower daemon using the package manager. For example, on Ubuntu/Debian, you can run the following command:sudoapt-getinstallupowerThis command installs the upower package, which provides the UPower daemon.After installing the UPower daemon, start the service using the following command:sudosystemctlstartupower.serviceVerify that the UPower daemon is now running by checking the status using the following command:systemctlstatusupower.serviceRestart your Flask application and try again. The error message should no longer appear.Note that the specific steps for installing and starting the UPower daemon may vary depending on your system and distribution. You may need to consult the documentation or package manager for your system to find the appropriate package and commands to install and start the UPower daemon.Error: Ubuntu: SSL: Handshake failed error when browsing local web site in google-chrome from UbuntuProblem:  When browsinghttps://127.0.0.1:5000receive error: SSL: Handshake failed. when logged into ZScaler[39102:39117:0503/091443.791015:ERROR:ssl_client_socket_impl.cc(992)] handshake failed; returned -1, SSL error code 1, net_error -202
[39102:39117:0503/091447.197078:ERROR:ssl_client_socket_impl.cc(992)] handshake failed; returned -1, SSL error code 1, net_error -202
[39102:39117:0503/091447.998238:ERROR:ssl_client_socket_impl.cc(992)] handshake failed; returned -1, SSL error code 1, net_error -202
[39062:39086:0503/091448.177119:ERROR:cert_issuer_source_aia.cc(134)] AiaRequest::OnFetchCompleted got error -301
[39062:39086:0503/091448.178211:ERROR:cert_issuer_source_aia.cc(134)] AiaRequest::OnFetchCompleted got error -301
[42678:42704:0503/093402.088689:ERROR:cert_verify_proc_builtin.cc(677)] CertVerifyProcBuiltin for clientservices.googleapis.com failed:
----- Certificate i=2 (CN=NCA-DPI1,OU=ITSO,O=Centers for Disease Control and Prevention,L=Atlanta,ST=Georgia,C=US) -----
ERROR: No matching issuer found[39062:39086:0503/091448.178351:ERROR:cert_verify_proc_builtin.cc(677)] CertVerifyProcBuiltin for optimizationguide-pa.googleapis.com failed:
----- Certificate i=2 (CN=NCA-DPI1,OU=ITSO,O=Centers for Disease Control and Prevention,L=Atlanta,ST=Georgia,C=US) -----
ERROR: No matching issuer foundSolution:Verify that your Python installation has a valid CA bundle that can be used to verify the SSL/TLS certificate. You can do this by running the following command in your terminal:python-c"import ssl; print(ssl.get_default_verify_paths())"This should print out the paths to the CA bundle and the OpenSSL configuration file used by Python. If the paths are not found or are invalid, you may need to update your Python installation or install a valid CA bundle.Path should return/usr/lib/ssl/certsDownload the missing CA certificate from the server that issued the certificate. You can use the openssl s_client command to retrieve the certificate, like this:cd$HOMEopenssls_client-showcerts-connectclientservices.googleapis.com:443</dev/null2>/dev/null|opensslx509-outformPEM>clientservices.googleapis.com.crt
openssls_client-showcerts-connectoptimizationguide-pa.googleapis.com:443</dev/null2>/dev/null|opensslx509-outformPEM>optimizationguide-pa.googleapis.com.crt
openssls_client-showcerts-connectsafebrowsing.googleapis.com.com:443</dev/null2>/dev/null|opensslx509-outformPEM>safebrowsing.googleapis.com.crt
openssls_client-showcerts-connectaccounts.google.com:443</dev/null2>/dev/null|opensslx509-outformPEM>accounts.google.com.crt
openssls_client-showcerts-connectupdate.googleapis.com:443</dev/null2>/dev/null|opensslx509-outformPEM>update.googleapis.com.crt
openssls_client-showcerts-connectwww.google.com:443</dev/null2>/dev/null|opensslx509-outformPEM>www.google.com.crtdns.googleInstall the CA certificate in your system's trusted CA store by copying it to the /usr/local/share/ca-certificates/ directory:sudocpclientservices.googleapis.com.crt/usr/local/share/ca-certificates/
sudocpoptimizationguide-pa.googleapis.com.crt/usr/local/share/ca-certificates/
sudocpsafebrowsing.googleapis.com.crt/usr/local/share/ca-certificates/
sudocpaccounts.google.com.crt/usr/local/share/ca-certificates/
sudocpupdate.googleapis.com.crt/usr/local/share/ca-certificates/
sudocpwww.google.com.crt/usr/local/share/ca-certificates/Update the CA bundle by running the following command:sudoupdate-ca-certificatesError: Ubuntu: CURL: SSL: Certificate problem: unable to get local issuer certificate when installing dockerSymptom:  Receive error: curl: (60) SSL certificate problem: unable to get local issuer certificate when installing dockerReference:Stack OverflowDetails:The problem may be related to the way the firewall is handling certificates. The certificate of the firewall may be untrusted/unknown from within the wsl environment.Resolution:Option 1:If logged into zscaler, log outTry AgainOption 2:Export the firewall certificate.Export the firewall certificate from the windows certmanager (certmgr.msc).The certificate may be located at "Trusted Root Certification Authorities\Certifiactes"Export the certificate ZScaler Root CA as a DER coded x.509 and save it under e.g. "_templates/certs/zscaler.cer".Configure WSL.open wsl terminal from VS Code : should default to Ubuntu: directory "/Users/zfi4/OneDrive - CDC/DAVT Analytics/davt-dev-jcbowyer"wsl
opensslx509-informDER-in_templates/certs/zscaler.cer-out./zscaler.crt
sudocpzscaler.crt/usr/local/share/ca-certificates/
sudoupdate-ca-certificatesError: Ubuntu: Python installation error: ImportError: No module named apt_pkgProblem: Python installation error: ImportError: No module named apt_pkg occurs when running sudo apt-get updateReference:Stack OverflowSolution:```sh
sudo apt remove python3-apt
sudo apt autoremove
sudo apt autoclean
sudo apt install python3-apt
```Error: Ubuntu: NPM can't find module "semver" error in Ubuntu 19.04Problem: NPM can't find module "semver" error in Ubuntu 19.04 when installing nodejs upgradeReference:AskUbuntuSolution:Run the following commands to uninstall node:cd$HOMEsudorm-rf/usr/local/bin/npm/usr/local/share/man/man1/node*~/.npm
sudorm-rf/usr/local/lib/node*
sudorm-rf/usr/local/bin/node*
sudorm-rf/usr/local/include/node*
sudoapt-getpurgenodejsnpm
sudoaptautoremoveRun the following commands to reinstall node:workonOCIO_PADE_DEVcd$VIRTUAL_ENVrm-rf/usr/local/bin/npm/usr/local/share/man/man1/node*~/.npm
rm-rf/usr/local/lib/node*
rm-rf/usr/local/bin/node*
rm-rf/usr/local/include/node*# install node with nvm - node version manager# wget https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh# chmod +x install.sh# ./install.sh# source ~/.bashrc# nvm list-remote# install node with wgetwgethttps://nodejs.org/dist/v18.12.1/node-v18.12.1-linux-x64.tar.xz
tar-xfnode-v18.12.1-linux-x64.tar.xz
rsync-avnode-v18.12.1-linux-x64/bin/*./bin
rsync-avnode-v18.12.1-linux-x64/lib/node_modules/./lib/node_modules/cd$VIRTUAL_ENV/binhash-r# Verify installation usingnode-v
npm-v

npminstallpuppeteer
npminstallnpm@9.1.2
npminstall@mermaid-js/mermaid-cli

sudoapt-getupdate
sudoapt-getinstall-ylibgbm-devError: VSCode: End of line character is invalidSymptom: Receiving error: "End of line character is invalid"Reference:Boot.dev BlogSolution for One File:Unless you work on a Windows-only team, the answer is almost always to change all your code to the Unix default of LF.Open VS Code and go the the bottom right of the screen in VS CodeClick the button that says LF or CRLF and change to LF (most all cases)Soulution for All New Files:For Workspace TabOpen VS CodeOpen File Menu > Preferences > SettingsSearch for CRLFChange Files: EOL Setting to LFFor User TabOpen VS CodeOpen File Menu > Preferences > SettingsSearch for CRLFChange Files: EOL Setting to LFSolution for All Existing FilesDownload VS Code Exntension:Change LF to CRLFProblem: WSL: Github clone fails: "fatal: unable to access 'https://github.com/cdcent/data-ecosystem-services.git': Could not resolve host: github.com':"Symptom: Trying to clone a github repository and receive error: "fatal: unable to access Could not resolve host: github.com "Reference:Stack OverflowResolution:Option 1:Restart WifiOption 2:Run terminal command in WSL:gitconfig--global--unsethttps.proxyRestart Terminal in WSL:Option 3:Log Off ZScalerTry againOption 4:Log Back into ZScalerTry againOption 5:Fix wrong/empty /etc/resolv.conf file.To view contents of /etc/resolv.conf file:code/etc/resolv.confTo fix:sudorm/etc/resolv.conf
sudobash-c'echo "nameserver 8.8.8.8" > /etc/resolv.conf'sudobash-c'echo "[network]" > /etc/wsl.conf'sudobash-c'echo "generateResolvConf = false" >> /etc/wsl.conf'sudochattr+i/etc/resolv.confProblem: WSL: "Logon failure: the user has not been granted the requested logon type at this computer."Symptom: Trying to connect to WSL from Windows Terminal and receive error: "Logon failure: the user has not been granted the requested logon type at this computer."Reference:Stack OverflowSolution:Open Powershell as AdministratorRun the following command:Get-Servicevmcompute|Restart-ServiceProblem: WSL: Windows: Shell Script keeps exiting unexpectedlySymptom:  WSL windows keeps exiting unexpectedlyReference:SuperUserDetails:Examine your wsl -l -v outputIt may show that the docker-desktop-data instance got set as the default when you uninstalled Ubuntu.Docker-desktop-data is not a bootable instance, since it has no /init in it.Resolution with Powershell or CMD:wsl--set-defaultUbuntu-20.04