fabiochiu/medium-articles huggingface dataset:
Turn your meh Notebook (.ipynb) into a pretty PDF on MacOS
By: Jack Lu
Tags: Pdf, Jupyter Notebook, Python, Mac

For me, storage on my mac is precious and limited and expensive.

I opted to take the other suggestion in her post which was to install Pandoc, which was only part of the requirement to make this conversion work. To keep the story of my travails brief, I got errors on my terminal, which led me to Stackoverflow and other sites, which led me to installing a bunch more things.

I am going to help you avoid my errors (you’re welcome 😃), and get you that sleek PDF from a Jupyter Notebook that you might want to send to someone, somewhere relatively unimportant.

Just follow my instructions and you will be good to go!

Packages to Download:

Install nbconvert Install Pandoc on MacOS Install XeLaTeX on MacOS Run commands

Install nbconvert

This part is criticial. If you don’t already have nbconvert installed, what are you even doing here?

Anyway, please visit and rectify your error right away: https://nbconvert.readthedocs.io/en/latest/install.html

Or run this on your mac terminal:

$ pip (pip3) install nbconvert



# OR



$ conda install nbconvert

Install Pandoc

Next let’s install pandoc, please visit: http://www.texts.io/support/0003/. You will see a page that looks like image below.

Alternative site to download pandoc: https://pandoc.org/installing.html

Install XeLaTex

Visit: http://www.texts.io/support/0001/. You will a page that looks like below.

Commands

At this point you will need to run a set of commands on your Mac terminal and install additional packages.

First, tlmgr — the name of a package and configuration manager.

$ sudo tlmgr update --self

Next, install a list of dependencies for tlmgr. Yes, copy and install all of it. Just do it!

$ sudo tlmgr install tcolorbox pgf xcolor environ trimspaces mathpazo parskip adjustbox collectbox eurosym ecs ucs enumitem ulem mathrsfs jknapltx rsfs collection-fontsrecommended

If a dependency is missing (you can never know!), check the error message on your terminal and install that one as well.

$ sudo tlmgr install <missing dependency>

FINALLY, convert your boring (.ipynb) to (.pdf)

Ensure that you’re in the same directory same the notebook before running the command:

$ jupyter nbconvert --to pdf your_notebook.ipynb

Path Issue

If you get a path error “500: Internal Server Error, nbconvert failed: xelatex not found in PATH”, run this command:

export PATH=/Library/TeX/texbin:$PATH

REFERENCES
Jupyter notebook useful commands and keyboard shortcuts
By: Rajat Jain
Tags: Command Line, Keyboard Shortcuts, Coding, Beginner, Jupyter Notebook

If there are shortcuts available, why to go through long ways. Just use them and save time, increase productivity, and most importantly build up confidence about things.

Absolutely it will make you feel confident to work on the Jupyter notebook.

Before understanding the keyboard shortcuts, need to understand Command mode and Edit mode.

Edit mode: As the name itself suggest, when you are writing (code/markdown/heading) in one of the cells within the notebook it’s in edit mode. Another way to know about it with the color of the cell. if it’s Green then in edit mode.

Command mode: when you’re not writing or editing anything inside the cell, it’s in command mode. And if the cell is in Blue color then it’s in command mode.

Keyboard shortcut to switch between Command mode and Edit mode

Esc to go into command mode for the cell you selected.

to go into command mode for the cell you selected. Enter to go back to Edit mode for the cell you selected.

Command mode shortcuts:

Now let’s see the keyboard shortcuts for the command mode. Though it’s not a complete list of the shortcuts but are the most useful ones.

m: Convert a cell to a markdown cell

Convert a cell to a markdown cell y: Convert a cell back to a code cell

Convert a cell back to a code cell a: Insert new cell above to current one

Insert new cell above to current one b: To insert new cell below to current one

To insert new cell below to current one d + d: Delete the current cell

Delete the current cell z: Undo the last operation

Undo the last operation f: To find and replace the word

To find and replace the word Shift + Up/Down: Select multiple cells

Select multiple cells Space: Scroll notebook downwards

Scroll notebook downwards Shift + Space: Scroll notebook upwards

Edit mode shortcuts:
Themes for Jupyter Notebooks
Tags: Data Science, Jupyter Notebook

Install the jupyter notebook theme packages

pip install jupyterthemes

or, if it is in Google colab

!pip install jupyterthemes

After successfully installing the theme package execute following command to list down all the available themes
03 Eigen Values and Eigen Vectors
By: Meeraj Kanaparthi
Tags: Videos, Eigenvalue, Machine Learning, Jupyter Notebook, Eigenvectors

In linear algebra, an eigenvector (/ˈaɪɡənˌvɛktər/) or characteristic vector of a linear transformation is a nonzero vector that changes by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted by {\displaystyle \lambda }, [1] is the factor by which the eigenvector is scaled.

Geometrically, an eigenvector, corresponding to a real nonzero eigenvalue, points in a direction in which it is stretched by the transformation and the eigenvalue is the factor by which it is stretched. If the eigenvalue is negative, the direction is reversed.[2] Loosely speaking, in a multidimensional vector space, the eigenvector is not rotated.

English

हिंदी (Hindi)

తెలుగు (Telugu)

Code:

Social Media:

https://www.linkedin.com/in/meeraj-k-69ba76189/

https://facebook.com/meeraj.k.35

https://www.instagram.com/meeraj.kanaparthi1/

https://twitter.com/MeerajKanapart1
Show me your dots and I’ll tell you your future…………
By: Richard Pelgrim
Tags: Climate Change, Data Science, Conflict, Jupyter Notebook

I’ve got some first results to show! Very early days — I’m still mostly wrangling my datasets into shape — but I’ve got some maps; and as well all know, where there are maps there’s a good chance there will be………………

…dots. LOTS of dots.

Three-hundred-and-forty-thousand-four-hundred-and-sixty-nine dots, to be exact. The blue dots are all the GHCN weather stations; the orange ones are the UCDP conflict incidents. Even just eyeballing this, it’s clear that some geographical areas have extremely dense weather measurement coverage (US, Northern Europe, Australia) while in other areas the GHCN coverage is much more sparse. It’s not a total surprise that many of the densest conflict areas have little weather station coverage.

Wrangling Strategy #1: Spatial Join THAT!

To narrow down my areas of interest I performed an inner spatial join on the two datasets. Since both datasets have exact latitude and longitude points for each observation, a direct inner spatial join would reveal only conflict incidents and weather stations that share an exact location. While this could potentially lead to an interesting discovery of guerrilla tactics for occupying weather stations, that is (unfortunately) not the main aim of this project.

So instead, I used the geopandas package to create a buffer of 100km around the conflict incident observations. This outputs a polygon geometry which — when specifying the correct ‘within’ inner join — can be used to find points in a second dataset that lie within these polygons:

Technical side-note: in order to be able to execute the .buffer method the Coordinate Reference System (CRS) of your geodataframe needs to be set to EPSG3857 (in meters rather than degrees), using the .to_crs method. My Jupyter Notebook kernel kept dying while trying to execute that method and it took me more than a week (no kidding) to figure out what was going on. You can retrace some of my rabbit-hole steps here and here.

But the hours spent trudging through forums and re-installing my conda packages and environments were SO WORTH IT, BABY!

Wrangling Strategy #2: Custom Extraction Function

As mentioned earlier, the GHCN dataset is too big to extract entirely onto my computer. So my second wrangling strategy has been to write a function that takes a conflict incident ID, finds all weather stations within a specified radius of that conflict incident, extracts .dly files containing the weather data into my notebook as a dataframe, and finally saves those dataframes locally as .csv files. I’ve done this in a separate notebook (‘01_Data_Collection’) so that this time-intensive process only has to run whenever I need to access new station files. In my actual wrangling notebook, I can then directly open the .csv files stored locally.

For the full code, check out the project’s Jupyter Notebooks on my Github.
Every Data Scientist Needs This
By: Parisa Daeijavad
Tags: Github, Jupyter Notebook

Sync your ipynb file with Github

An IPYNB file is a notebook document used by Jupyter Notebook, an interactive computational environment designed to help scientists work with the Python language and their data. It includes the inputs and outputs of computations, mathematics, machine learning, images, and more.

For pushing your first project to Github you need to:

Open a web browser. Navigate to github.com and create an account if you haven’t already. Navigate your way to a new repository, fill out the form, and follow the instructions displayed there. After creating the new repository, you will be taken to the project homepage. Uploading your file (.ipynb Jupyter notebook) to the repository, if you have that already installed, and go to the directory containing the project.

When you’re done, you should see your files here: https://github.com/user/your-repository

To push new changes in your project to Github:

Go to Google Colab → colab.research.google.com Under the Github tab, enter your GitHub URL, which is https://github.com/user/your-repository in this case. Hit the search button and choose your .ipynb file to open it in Colab. Once you change your code, you need to push it to Github. To do so, go to File → save a copy in Github. Change the branch if you want then click ok.

When you’re done, you should see a new commit under history in your files here: https://github.com/user/your-repository/commits/main/Name.ipynb
Waiting for Long time for a Jupyter cell to complete the execution?
By: Pratik Erande
Tags: Time Saving, Data Science, Long Process, Model Training, Jupyter Notebook

Do you run long processes in Jupyter notebooks and visit them periodically to check if the job is finished?

Yes..?

Perfect, then I have a small idea to help you. (You can skip the paragraphs below and jump to Idea! section if you want to save time)

Thanks to my project team, they gave me large compute power machines to run my Jupyter notebooks. Now, it’s my responsibility to use the instance to the fullest and shut it down when my experiments are over.

Often when I am training something it takes long time, and I start doing something else during that time on my local machine. Usually I need to periodically open my jupyter tab to see if the process is finished, I do not like interruptions plus sometimes the process is finished and the instance seats idle (wasting our money). I wanted something to tell me exactly when the process finishes..

Idea!

We can simply play music at the end of our any Long Running process in Jupyter.

import IPython

IPython.display.Audio(“your_music_file.mp3”,autoplay=True)

Have fun ! :D
02 Covariance Matrix
By: Meeraj Kanaparthi
Tags: Covariance Matrix, Datascience Training, Vidoes, Machine Learning, Jupyter Notebook

In probability theory and statistics, a covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix, or variance–covariance matrix) is a square matrix giving the covariance between each pair of elements of a given random vector. Any covariance matrix is symmetric and positive semi-definite, and its main diagonal contains variances (i.e., the covariance of each element with itself).

Below is the demo in English, हिंदी (Hindi), తెలుగు (Telugu)

English

हिंदी (Hindi)

తెలుగు (Telugu)

Code:

Social Media:

https://www.linkedin.com/in/meeraj-k-69ba76189/

https://facebook.com/meeraj.k.35

https://www.instagram.com/meeraj.kanaparthi1/

https://twitter.com/MeerajKanapart1
04 Singular Value Decomposition
By: Meeraj Kanaparthi
Tags: Datascience Training, Videos, Machine Learning, Svd, Jupyter Notebook

In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix that generalizes the eigendecomposition of a square normal matrix to any {\displaystyle m\times n} matrix via an extension of the polar decomposition.

Below is the demo in English, हिंदी (Hindi), తెలుగు(Telugu)

English

हिंदी (Hindi)

తెలుగు (Telugu)

reduction: https://en.wikipedia.org/wiki/Gaussian_elimination

Example : https://www.d.umn.edu/~mhampton/m4326svd_example.pdf

Social Media:

https://www.linkedin.com/in/meeraj-k-69ba76189/

https://facebook.com/meeraj.k.35

https://www.instagram.com/meeraj.kanaparthi1/

https://twitter.com/MeerajKanapart1
01 Mean
By: Meeraj Kanaparthi
Tags: Demo, Colab, Machine Learning, Mean, Jupyter Notebook

There are several kinds of mean in mathematics, especially in statistics.

For a data set, the arithmetic mean, also called the expected value or average, is the central value of a discrete set of numbers: specifically, the sum of the values divided by the number of values. The arithmetic mean of a set of numbers x1, x2, …, xn

Finding mean is a very important concept in Machine Learning. Below is the demo for finding mean. Below is the demo in English, हिंदी (Hindi), తెలుగు (Telugu)

English

हिंदी (Hindi)

తెలుగు (Telugu)

colab: https://colab.research.google.com/gist/kmeeraj/2e59c8b1ebafab05984195247ee24c78/01-mean.ipynb

Reference: https://en.wikipedia.org/wiki/Mean

github: https://github.com/kmeeraj/machinelearning/tree/develop

github demo: https://github.com/kmeeraj/machinelearning/blob/develop/basics/mean/Column%20and%20Row%20Vector%20Mean.ipynb

Social Media:

https://www.linkedin.com/in/meeraj-k-69ba76189/

https://facebook.com/meeraj.k.35

https://www.instagram.com/meeraj.kanaparthi1/

https://twitter.com/MeerajKanapart1
Show Your .NET Code Effectively With Jupyter Notebook
By: Jung-Hyun Nam
Tags: Csharp, Developer, Jupyter Notebook, Powershell

You can show your .NET code as a presentation with Jupyter Notebook, .NET Interactive.

The precious .NET 5 has finally arrived in our workspace, and I’m thrilled. .NET 5 is a unified and cross-platform supported runtime for C# and F# developers. Also, it has balanced cross-platform features across all environments, including existing your old familiar Windows desktop environment and Linux, Cloud-native, macOS, even machine learning, Apache Spark, and Web Assembly.

I’m recently taking a series of the episode called the Azure Playlists project in cooperation with Microsoft Korea. I thought a lot about how to present my PowerShell code to viewers effectively. After many trials and errors, I got a neat but simple idea.

With the release of .NET 5, Microsoft also launches many useful tools as open-source projects. .NET Interactive is one-of-them, and I found it.

.NET Interactive provides three types of .NET kernels, C#, F#, and PowerShell. I used PowerShell to introduce Azure related tools. Because PowerShell works almost the same across all three OSes (including Windows, Linux, and macOS). With .NET Interactive, I can reuse my existing knowledge to Jupyter notebook without any hassle.

C#, F#, and PowerShell supported!

As you already know, the Jupyter notebook can contain the runnable code and its results to understand its meaning and how it works. This time, you can replace your code with C#, F#, or PowerShell.
Performing Analysis Of Meteorological Data
By: Atif Ateeque Kundlik
Tags: Data Visualization, Data Analyst, Python, Data Science, Jupyter Notebook

I am very pleased to share my first internship project which is based on

Performing Analysis of Meteorological Data Project of Data Analytics using Python Domain.

OBJECTIVE:

To transform the raw data into information and then convert it into knowledge. We will be performing some basic tasks to perform our analysis such as data cleaning, testing the hypothesis and finally concluding certain information.

Null Hypothesis (H0)

“Has the apparent temperature and humidity compared monthly across 10 years of the data, indicate an increase due to Global warming.”

The H0 means we need to find whether the average Apparent temperature for the month of a month say April starting from 2006 to 2016 and the average humidity for the same period have increased or not. This monthly analysis has to be done for all 12 months over the 10 year period. So you are basically resampling your data from hourly to monthly, then comparing the same month over the 10 year period.

DATASET:

The dataset has hourly temperature recorded for last 10 years starting from 2006–04–01 00:00:00.000 +0200 to 2016–09–09 23:00:00.000 +0200. It corresponds to Finland, a country in the Northern Europe.

Download from https://www.kaggle.com/muthuj7/weather-dataset

IMPLEMENTATION:

1.Importing all the required libraries

2. Loading data: Let’s first load the required weather dataset using pandas read CSV function.

3.Explore Data:

Check the shape of data using shape method. After that If we want to see what all the data types are in a dataframe, use dtypes

4.Data cleaning

4.1.Checking null value present in dataset or not

4.2. Change the format of data for better analysis

Change Formatted Date column type from object into date time . For this we use to_ datetime () function.

5. Describing the data

6. For better analysis we need to resample our data in monthly period of time

Data visualization:

PLOTS: Plot of variation in Apparent Temperature and Humidity with time

We need to find whether the for the month of a month say April starting from 2006 to 2016 and the for the same period have increased or not

Plotting the variation of humidity and temperature for April Months

Plots of Humidity and Temperature for all 12 months

Conclusion :

Average Apparent Temperature increase for year 2008–09 and then again decrease from 2009–10 to its average level. Then slightly increase for period 2010–11 and then drop again for year 2011–12. For year 2014–15 it decrease drastically and then come to it average level for year 2015–16.

It observed, for year 2009 the Average Apparent Temperature is at Highest 14.26(C) and for year 2015 the Average Apparent Temperature is at lowest 10.63(C).

There is no any change in Average Humidity observed. The humidity graph line is approximately parallel to the X-axis.

I am thankful to mentors at https://internship.suvenconsultants.com for providing problem statements and giving many of us a Coding Internship Experience. Thank you www.suvenconsultants.com
Why you should start small
Tags: Data Science, Big Data, Power Bi, Google Data Studio, Jupyter Notebook

Why you should start small

Photo by Matt Duncan on Unsplash

In a time where everyone is talking about Big Data, Data Science, Machine Learning and how you can build up new data driven products or even dream about to create a new unicorn — often you just need some simple dashboard skills.

One thing ahead

When you have a lot of customer data and your business is for example an big online shop you will definitely have a high chance of using machine learning and other related fields to develop better decision models, predict customer churns and have preciser forecasts. The latter is also for financial and controlling departments an interesting topic. Companies in the field of IoT will need Big Data technologies to handle the vast amount of data — no question about it — but you should not dream about all these interesting use cases if you don’t have the capabilities for it.

Reasons for starting small

There are many possible reasons for this circumstance:

You just don’t have enough or interesting data and/or systems.

You don’t have a use case (You should never do Data Science just because everyone is doing so).

You don’t have done your homework — e.g. maybe your company should first care about technical debts (stable data integration, a solid and scalable analytic platform, build up data science know how, etc. — read here more about it)

In this case you should think of simpler but still interesting and useful applications that you can approach with your data.

How to get started

Often all you need is a dashboard tool. A few great examples for such tools, that are also free are:

Google Data Studio or Google Sheets

Power BI Desktop

QlikSense Desktop

Dashboard with Data Studio — Image by Author

In addition, you will need an idea how to visualize data in a way that is preserved as simple, correct and intuitive.

I just had an experience where a bubble chart helped the management to understand the data way faster and to use it for important management decisions. So what I wanna say is that you should not dream to big if you just don’t have the possibilities (yet). Even small amounts of data and just simple dashboards technics can lead to significant insights. My advice: Get started with one of the tools listed above and work with your data.

A great book I can recommend is: Storytelling with Data: A Data Visualization Guide for Business from Cole Nussbaumer Knaflic and this blog about Google Data Studio.

For people who are already experienced with programming you might use tools like Jupyter Notebook to visualize your data. You can start right away with your browser at: https://jupyter.org/try.

Conclusion

I hope this brief article will help you to get started with working with your data and dreaming not to big (yet). In certain businesses especially small to medium sized ones you maybe will never need Data Science, Big Data and co. But even visualizing small amounts of data in a nice dashboard also can lead to better business decisions and interested readers.

Sources and Further Readings
Hands on Machine Learning to program in Jupyter notebook
By: Meeraj Kanaparthi
Tags: Python, Ipynb, Colab, Machine Learning, Jupyter Notebook

I created YouTube videos on Machine Learning using Jupyter notebook.

Hands on Machine Learning to program in jupyter notebook and easy to learn. This is my youtube channel for data science and Machine Learning for Beginners in English, हिंदी (Hindi) and తెలుగు (Telugu)

I have covered video tutorials in three languages, viz., English, Hindi and Telugu.

Following are playlist in English:

Promotional: https://www.youtube.com/watch?v=OaXZieKMqWQ&list=PL01e4bnNRw-Sl9Xt4bw1hmCfMBatVR7YH Basics in English: https://www.youtube.com/watch?v=TAHr-dLsDy8&list=PL01e4bnNRw-S71tN3OJGEMbvb9blR733G Machine Learning algorithms: https://www.youtube.com/watch?v=Zyo9lTvhAxk&list=PL01e4bnNRw-SZvEaIxSOo44Ip4ZPzLYvh

Following are playlist in Hindi:

Promotional: https://www.youtube.com/watch?v=GcHVwuOpubc&list=PL01e4bnNRw-RUmsUDf-X7wK5l85AwAKBx Basics in Hindi: https://www.youtube.com/watch?v=TGvSkbYb8oA&list=PL01e4bnNRw-TyOmg_LHziwfa0HVfqOF9J Machine Learning algorithms: https://www.youtube.com/watch?v=aaarXiwDnyk&list=PL01e4bnNRw-Rx9paxRUOheI2X8tIf7nJ5

Following are playlist in Telugu
JupyterLab Series
By: Roman Orac
Tags: Jupyterlab, Jupyter, Data Science, Analysis, Jupyter Notebook

A curated list of articles I wrote about JupyterLab.

Here are a few links that might interest you:

Some of the links above are affiliate links and if you go through them to make a purchase I’ll earn a commission. Keep in mind that I link courses because of their quality and not because of the commission I receive from your purchases.

A list of articles I wrote about JupyterLab

Before you go

Follow me on Twitter, where I regularly tweet about Data Science and Machine Learning.
Machine Learning and AI for All
By: Meeraj Kanaparthi
Tags: Neural Networks, Programing Tutorials, AI, Machine Learning, Jupyter Notebook

I do research in Machine Learning and AI algorithms. I like to share my programing tutorials for all in my channel https://www.youtube.com/channel/UCtcEuGgCTWzzXBGvwCKm5VA

Following Topics are covered:

01 Mean

02 Covariance Matrix

03 Eigen Values and Eigen Vectors

04 Covariance Matrix

05 Singular Value Decomposition (SVD)

06 Principal Component Analysis (PCA)

07 Loss Function
Organise your data analytics workflow
By: Daniel Quek
Tags: Workflow, Data Analytics, Jupyter Notebook, Data Preparation, Template

Overview

Ever wonder why your data analytics (DA) projects are getting messier and messier as each day goes by and is there a standard workflow to organise your DA projects? I have this problem when I started my DA journey and fortunately, I have found this workflow which works best for me. You can try this and maybe you will like it 😃

Step 1: Create folders

Every DA projects will always start with these folders

Create the above folders and put your files accordingly. The folder starts with a numeric digits so that it is easy to sort them by folder name. If you are going to dump every files into a general folder, it will be messy over time. Others will also have a hard time navigating your project.

0. Raw Data

This folder consists of your original datasets without any processing done. It can be either in .csv or .xlsx format.

1. Scripts

This folder will store your python or R scripts and the scripts will read from files stored in the 0. Raw Data folder.

2. Processed Data

All your processed data generated by your scripts will be stored in this folder.

3. Dashboards

Graphs, charts or tableau files can be stored in this folder.

4. Documents

Other documents like data dictionary or literature reviews can be placed here.

Step 2: Create a data dictionary
Why put your ML results into a dashboard
By: Huifang Yeo
Tags: Dashboarding, Time Series Analysis, Technology, Machine Learning, Jupyter Notebook

Why put your ML results into a dashboard

While trying to perform a time-series analysis, there are many things to check. For instance, the above GIF shows me trying to scroll through the Matplotlib charts for each time-series for stationarity checks.

In the above use case, I’m evaluating 12 feature time-series for 12 cryptocurrencies and that amounts to 144 charts. Perhaps I am a little too ambitious to run the test for all the cryptocurrencies in one go. Then again I don’t see why I shouldn’t, since the steps for the analysis are pretty much the same.

What would you do in this case? Would you run the notebook repeatedly or duplicate it once for each cryptocurrency? I chose to load my data into atoti and visualize the time-series using interactive dashboards.

Now I am able to put together my analysis within a common view and filter by the cryptocurrency that I want to dive in. Did I mention that I also loaded each differencing of the time-series into the cube?

It’s interesting to see how the time-series becomes stationary with each differencing, represented by a different colored trace for each of the feature’s plot in the above GIF.

As a next step to my analysis, I ran the Granger causality test on the time-series for 50 lags to see which features can Granger-cause the returns of the cryptocurrency. Check out below a dashboard that I’ve built to quickly identify the features that I will use in my time-series forecasting for each cryptocurrency.

I can run and save as many lags of the test results into atoti and apply filters and color coding the cells (in green above) to see the lags that were rejected. This means that I do not have to discard my test results prematurely.

By using the Quick page filter widget that comes with atoti, I can apply filters easily on the page and toggle around the cryptocurrencies compare the patterns revealed by the statistical analysis.

Lastly, I personally preferred the atoti’s charts in a dashboard over Matplotlib charts in a Jupyter notebook, namely due to the interactive aspect. I can zoom in literally on the charts that are of interest and mouse over to see the relevant values.

In any case, I can always output my data from the atoti cube when needed by:

exporting tables to csv

downloading plots as png

programmatically querying from the cube into a Pandas dataframe

Check out my notebook and article on how I forecast the returns of different cryptocurrencies with Twitter attributes.

Do share with me on how you manage your machine learning results!
Data Visualisasi dengan Matplotlib
By: Kenisya Fudika
Tags: Data Visualization, Data Science, Jupyter Notebook, Datascience Training

Hi! I am kenisya. I like to share my new journey. I interested to design and data. Hope my article can helps you to solve your problem. Thank you ^^
How to prevent auto activate and show (base) after installing Anaconda.
By: Grassroot Engineer
Tags: Anaconda, Jupyter Notebook

Electronics engineer who is interested in CODING and believe in EFFORT. — formerly http://grassrootengineer.medium.com
3 Ways to Add Images to Your Jupyter Notebooks
By: Esmaeil Alizadeh
Tags: Python, Jupyter, Data Science, Programming, Jupyter Notebook

Approach 3: Embed an Image by Base64 Encode-Decode

The first two approaches rely on external resources. In Approach 1, we rely on a URL, and any change in the original link will impact the image in the notebook. In Approach 2, we use the path to a file that is saved locally. Any change in the filename or path may impact the image in the notebook. Unlike the previous methods, Approach 3 embeds the image as a text using the Base64 encoding algorithm. This way, we will not be relying on any external resources for the embedded image. Hence, we can have all images embedded in the same notebook file.

Base64 is a binary-to-text encoding algorithm to convert data (including but not limited to images) as plain text. It is one of the most popular binary-to-text encoding schemes (if not the most popular one). It’s widely used in text documents such as HTML, JavaScript, CSS, or XML scripts. However, technically speaking, you can even encode/decode audio or video files!

First, you need to encode your image. For this, you can use the online tool Base64-Image. After you upload your image, you can then click on “copy image,” as shown below:

Screenshot of the uploaded image at base64-image

Now you can paste the encoded image code into your notebook, but first, you should remove data:image/png;base64, at the beginning. Don't forget to also remove the comma after base64 !

Now that we have the encoded image code, we can use the Python standard base64 library to decode the base64 data, as shown below:

from IPython import display

from base64 import b64decode base64_data="iVBORw0KGgo.....<deleted to save space!>.....TkSuQmCC" display.Image(b64decode(base64_data))

As you may have noticed by now, the main advantage of using Base64 to add all images to your notebook is the fact that you no longer need to worry about any external resources for your images, as they are all self-contained in your notebook. The other point to be aware of is that including the images in your notebook will increase your notebook’s file size depending on the image resolution.
Getting Started with the Jupyter Notebook (Part 3)
By: Charles Simms
Tags: Jupyter Notebooks, Data Science, Data Analytics, Python Tutorial

DataFrame

The next major data structure in the Pandas package is the DataFrame , a 2-D array, where each column is basically a Series with a shared index. A DataFrame is an integral data structure that provides a mapping for various data formats and storage mechanisms. While a DataFrame can be created in myriad ways, most are beyond the scope of this tutorial. Please refer to the official documentation for a more comprehensive look at creating them.

Once we’ve familiarized ourselves with more advanced concepts, like the NumPy (Numeric Python) module, you'll learn other techniques to create and manipulate a DataFrame . In a future lesson, you'lllearn to create a DataFrame by reading data from a file or a website. For this notebook in Medium, we'll create a DataFrame by using a dictionary to map column labels to the values in the column, which we'll store in a list.

The next code cell creates a DataFrame by using a dictionary with four entries, where each entry contains a four element list. This create a DataFrame that has 16 entries, some of which are missing values (we're going to encode these using the Python None keyword, and Pandas translates this into NaN , which is typical shorthand for Not a Number). The last step is to display the resulting DataFrame , which has increasing integers as row labels (which were automatically assigned since we didn't supply them) and the dictionary key values as column labels.
Converting a Jupyter Notebook to Pdf
Tags: Ds4n6, Data Science, Jupyter Notebook, Forensics

Originally published at http://www.ds4n6.io.

jupyter nbconvert my_notebook.ipynb --to pdf

If you want to change the font size, for instance, you can do it through overriding templates ( docs, base templates), but it can get complex. So the easiest way is to convert to LaTeX first, do some simple replacements, and then convert the resulting LaTeX to pdf via the pdflatex utility:

$ jupyter nbconvert --to latex my_notebook.ipynb

$ sed -r -i 's/documentclass\[11pt\]\{article\}/documentclass[8pt]{extarticle}/' my_notebook.tex # <---- This modifies the font size from 11pt to 8pt

$ sed -r -i 's/geometry\{verbose,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}/geometry{verbose,tmargin=0.5in,bmargin=0.5in,lmargin=0.2in,rmargin=0.2in}/' my_notebook.tex # <---- This sets the margins

$ pdflatex my_notebook

$ rm my_notebook.tex; rm my_notebook.out; rm my_notebook.aux; rm Requirements:

Requirements:

texlive-xetex must be installed in the system (Ubuntu installation: apt-get install texlive-xetex texlive-generic-extra)

Refs:
Sharing Variables Between Jupyter Notebooks
Tags: Ds4n6, Data Science, Jupyter Notebook, Forensics

Originally published at http://www.ds4n6.io.

Often times you will be working in different notebooks (with different kernels) and you will want to share a variable from one notebook to the other.

In order to share a variable (e.g. string, dict, Series, DataFrame, etc.) just use the %store magic:

On the “sending” side:

%store mydf

On the “receiving” side:

%store -r mydf

That’s it!

Read more about the %store magic here:

https://ipython.readthedocs.io/en/stable/config/extensions/storemagic.html
[Project 001] 醫療對話-0
By: Yu-Wei Chen
Tags: Jupyternotebook, Terminal, Project001

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
pySpark Development environment setup
By: John Di Zhang
Tags: Pyspark, Jupyter Notebook, Data Engineering

1. Sample startup repository

https://github.com/zdjohn/spark-setup-workshop

2. Setup dev denpendency

python3 pip3

install tox pip install tox (ref: https://tox.readthedocs.io/en/latest/index.html)

tox is not just a testing tool. It helps you to set up a dev environment with common development library. of course, you can update/change based on your own preference.

run tox -e dev dev dependency is configured inside tox.ini file: https://github.com/zdjohn/spark-setup-workshop/blob/master/tox.ini

dev dependency is configured inside tox.ini file: https://github.com/zdjohn/spark-setup-workshop/blob/master/tox.ini source to tox dev virtual environment source .tox\\dev\\bin\\activate

3. Setup project dependency

note: please make sure your dev venv is now activated

install dependencies: pip install -r requirements.txt

add dev virtual environment to jupyter notebook python -m ipykernel install --user --name=pyspark-sample

4. spark test run

jupyter notebook

spark-submit --master local[*] --deploy-mode client helloworld.py

5. package your project
Python Básico — If & Else. Neste tutorial vamos aprender como…
By: Renato Dantas
Tags: Python, If Else, Python3, Lógica De Programação, Jupyter Notebook

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Glitch Free Anaconda Mac Installation In Two Steps
By: Adam Ross Nelson
Tags: Programming, Data Science, Mac, Anaconda, Jupyter Notebook

Image Credit: Adapted from Wikimedia Commons.

Glitch Free Anaconda Mac Installation In Two Steps

Update, February 2021, YouTube video tutorial available below.

Introduction

Whoa, the blogs and support forums are blowing up with users that are having trouble getting Anaconda to install on a Mac.

Anaconda Mac Installation that will (likely) be glitch free.

These two steps will help you avoid seeing: -bash: jupyter: command not found after installing Jupyter Notebook.

Step One

Download the command line installer. Here is what that looks like:

To install Anaconda, on a Mac, without glitching. Start by downloading the command line installer. Image credit: Screen capture prepared by @adamrossnelson.

Step Two

Go over to the terminal to find the name of your file (it changes over time as new versions release). Use the command: ls ~/downloads/Anaconda*.sh

Highlight and copy the output to your clipboard.

Then type in the terminal bash [insert output] Here is how that might look for you:

The blue text is user input (what you type as the user). The gray text is computer output. What the computer provides for you as you interact with the command line in the terminal.

Once you see the “Welcome to Anaconda3 . . .” you’re in good shape. Follow the on-screen prompts. Leave comments if this doesn’t work.
Jupyter Notebook on your iPad with Junno Connect
By: Erwin Schleier
Tags: Data Science, iPad, Jupyter Notebook

Apps you need:

Open Blink and navigate to your ssh key

Click on “Keys” than on your first key which should be “id_rsa” and click on “Copy Public Key”.

Close the window and ssh into your server.

ssh user@your-server-address

Add your ssh-key into .ssh/allowed_keys

sudo nano ~/.ssh/authorized_keys

Install jupyter notebook on it

pip install notebook

Create a jupyter notebook config file with the command

jupyter notebook --generate-config

Now edit the created file called jupyter_notebook_config.py

nano -/.jupyter/jupyter_notebook_config.py

Search for the line called c.NotebookApp.allow_remote_access and change the value to True.

c.NotebookApp.allow_remote_access = True

To ensure a communication between your iPad and server can be established you have to open the port where your jupyter notebook will be hosted. On default the port number is 8888.

sudo ufw allow 8888

Verify your open ports

sudo ufw status verbose

As you might not want to always connect to your server and run a jupyter session, it is more pleasant to start a background session on your server with Tmux.

# start a new Tmux session tmux new -s jupytersession jupyter notebook --no-browser

Now press Ctrl + B to detach from this session and then press D. This will send your Jupyter notebook instance to the background, and keeps running even if you disconnect from your server.

However, if you want to stop your Jupyter instance, reconnect to your server and type the command:

tmux a -t jupytersession

Now all server side settings are done and we are ready to open Junno Connect.

On the left sidebar of the home screen click on “Add Jupyter Server..”, where you will facing the following screen:

Description: Just a random name for your connection

Type: We will go with “Local port forwarding” as we want to establish a SSH connection.

Address: your server ip-address which you also used before to ssh into your server

Port: default value with 22 should be fine.

Type: Private key

User: username of your server which you used before to ssh into your server

Now we will import your private ssh key into Junno Connect, therefore go back to Blink and ensure that you are not connected on any server. Than type the command

That will open a configuration window of Blink.

Click on “Keys” than on your first key which should be “id_rsa” and click on “Copy Private Key”.

Go back to Junno Connect and continue your settings by clicking on “Set private key” and click the button “Import from Clipboard”. Finish the ssh-key setup by clicking on the top right corner “Save”.

Host: localhost

Port: 8888

Click on “Save” on the top right corner to save your settings. Ensure that your jupyter notebook application is running on the server. If so, establish a connection on Junno Connect!

If everything went well, you will see the familiar jupyter home screen. Additional packages can be easily installed by the in-build terminal or just switch to your server and do an usual pip install.
Announcing PrimeHub Community Edition v3
By: Gabriel Lo
Tags: Primehub, Data Science, Mlops, Machine Learning, Jupyter Notebook

Enabling your team of data scientists to effectively work together in an intuitive way

We are excited to announce our open-source, PrimeHub Community Edition (CE) v3 — a toil-free, multi-tenancy machine learning platform! This is our first major open source edition that we are releasing to the community and are so excited to share this with you. PrimeHub CE is outfitted with a brand new UI to support your teams’ productivity and compliments your intuitions when creating and running jobs in your Juptyter notebook.

Our mission for PrimeHub is to help take care of the clunky technical set-up tasks and operations so that your teams of data scientists can have a streamlined AI workflow. PrimeHub is built on Kubernetes and can be deployed on a private or public cloud. We do the heavy lifting by preparing a fast self-service environment in seconds, allowing an admin to manage users and resources per group, and also including a way to connect to various data sources while supporting several programming languages. With PrimeHub, your teams can collaborate on a single platform together rather than manning standalone work stations and save money by taking advantage of our collaboration model to setup, manage, and assign resources effectively.

Open-source is leading the rapid growth and innovation of the ML space, and we want to contribute to the groundwork of ML. We believe that PrimeHub CE can foster a community of innovation and make machine learning more approachable and democratized across industries.

PrimeHub CE v3 Highlights

Group-context selector

Group-context selector

Your view of PrimeHub is determined by whichever group-context you select to live in. Once you select your group in the top right corner, the work you do in your Jupyter notebook, along with all of the jobs you view, create, or update, will be in the context of the selected group.

To proceed working in another group, simply change your selection in the group-context selector.

SSH Server

Enable SSH Server in Advanced Settings

Access your self-launched JupyterHub via SSH. We provide diverse development options for people who like to use rich-feature IDEs, such as VSCode, and need to have a remote-SSH extension for remote development.

Admin Portal

Admin Portal link

All administration features such as managing your groups, resources, images, datasets and secrets, are in our Admin Portal that is accessible via the user avatar in the top right corner.
Jupyter Lab container for Kubeflow
By: Ivan N.
Tags: Kubeflow, Jupyter Notebook, Docker, Jupyterlab

How to get rid of Jupyter Notebook and use Jupyter Lab instead.

The Problem

We at DataSparQ love Kubeflow. But we don’t like the default Notebook options that are on the menu:

“You can have any colour you want, as long as it’s black”

They all come with:

Jupyter Notebook : which does not have dark mode or the ability to paste in the terminal

: which does not have dark mode or the ability to paste in the terminal Lack of (working) sudo , which comes apparent only after you realise apt get is the only way to install packages like ssh-keygen

, which comes apparent only after you realise is the only way to install packages like Default packages are fine if you need Tensorflow. Besides you can always install from your requirements file. (ok, maybe that one is the least annoying…)

are fine if you need Tensorflow. Besides you can always install from your requirements file. (ok, maybe that one is the least annoying…) Python 3.6; which is great, until you enter dependency limbo because you need that one package. And unlike the python packages, you can’t update the python version because of the lack of sudo.

The solution

Luckily for me, the lovely people behind Kubeflow have thought of the petty, dark-mode-liking developers like myself and have left us the ability to be the masters of our own containers. However, they did not want to make this job too easy by having up-to-date documentation.

Luckily for you, dear reader, I have already been through the minefield of Jupyter parameters and have found a working path. It is all available on GitHub:

Source Code

The Explanation

First a few warnings:

Photo by Ben Hershey on Unsplash

This container will run without any authentication and will have password-less sudo.

Here’s what comes in the box (container):

Python 3.8 ( or whichever is on the latest jupyter/scipy-notebook )

) Jupyter extensions: git, dask

Common terminal tools: ssh-keygen, vim, curl, jq

kubectl and helm (which is a very edge case use)

and (which is a very edge case use) A bash script that runs Jupyter lab with the “correct” settings

I encourage you to clone the repo and add/remove any packages you may need. Then it’s up to you to push that image to a repo that is accessible by your kubernetes cluster.

…or you can be lazy and paste ivndigint/kubeflow-jupyter:latest in the ‘Image’ field of the Kubeflow notebook window.

The best way to use this image (or its fork) is as a “BASE” layer and then add any project dependencies on top of it:

I have found this to be a good way to prevent any dependency mismatch between the different environments.
Save Jupiter Notebooks as PDF: An easy way
By: Kuldeep Pal
Tags: Machine Learning, Jupyter Notebook, Python, Towards Data Science

Install notebook-as-pdf 0.3.1

python -m pip install -U notebook-as-pdf

pyppeteer-install

How to use it?

Restart your notebook and PC and you fill find these option as in below pic

Create a notebook and then click “File -> Download As”. Click the new menu entry called “PDF via HTML”. Your notebook will be converted to a PDF on the fly and then downloaded.

Hope you liked this way.
How to Insert Images in Jupyter Notebook
By: Em Ekdahl
Tags: Python, Data Science, Technology, Programming, Jupyter Notebook

Sometimes it is just easier to insert a picture than to explain with text or code comments. This brief article shows you how to insert images into Jupyter Notebooks!

Gonzalez (2016)

Markdown

First, set your cell to markdown!

Local Images

Assuming your file structure looks like this:

/

+-- your_jupyter_notebook_file.ipynb

+-- img

+-- image_file_name_here.png

Insert your image like this:

![Title](img/ image_file_name_here .png)

Web Image

For web images, you can simply insert the url:
Simple Herb-Seasoned Turkey Breast
By: Keto Plans
Tags: Fitness, Bhfyp, Diet

This herb-seasoned turkey breast requires no fussy preparation. The result is a moist, tender entrée that the whole family will love.SLOW COOKER SIZE: LARGE



3 tablespoons olive oil1 boneless turkey breast, about 3 pounds1 medium onion, cut into wedges2 celery stalks, cut into large pieces1 teaspoon dried minced garlic1 teaspoon dried thyme1 tablespoon dried sage, crumbled1 teaspoon salt½ teaspoon ground black pepper½ cup water8 tablespoons butter, melted1.Rub the sides and bottom of the stoneware with 1 tablespoon of the olive oil.2.Heat the remaining 2 tablespoons olive oil in a large skillet over medium-high heat. Add the turkey and brown well on all sides, 8 to 12 minutes.3.Place the onion and celery in the slow cooker. Place the turkey on top. In a small bowl, combine the garlic, thyme, sage, salt, and pepper. Sprinkle the seasonings over the turkey breast.4.Pour the water around the turkey. Pour the melted butter on top of the seasonings. Cover and cook on Low for 7 to 9 hours.



PER SERVING: 384 CALORIES ★ FAT 27 G (63%) ★ NET CARBS 3 G (3%) ★ PROTEIN 31 G (32%)