fabiochiu/medium-articles huggingface dataset:
Realistic and Expectations
By: Elliot Wu
Tags: Diplomatic Relations, Military, Taiwan, USA, China

ENA Picture

Taipei, Dec. 21 (ENA)

The US president-elect Joe Biden will be inaugurated on January 20 as the 46th president of the United States. It is without any doubt that not only the US President but every member of the Senate and the Congress have an inseparable tie with Taiwan. How will the incoming Biden administration affect the circumstances in Taiwan and the cross-strait relations?

Since the People’s Liberation Army normalized its military aircrafts’ training mission and broke into Taiwan’s Air Defense Identification Zone (ADIZ) more often, Taiwanese people have increased their anger against China more than ever and take more attention to its national security matters.

The United States, a staunch ally which Taiwan puts tons of hopes expecting the international giant could help defend the island. However, no official statement of what Taiwanese people hope exactly (send troops) has been published. Instead, large numbers of arms sales have been made.

Some said that the recent sale of an iconic drone the UAV MQ-9 Reaper to Taiwan is the best symbol of the tight relation between the U.S. and Taiwan, and even considered the actual US President Donald Trump as “the most Taiwan friendly president in history”.

Indeed, the largest arms sales with several high firepower weapons included have been made during Mr. Trump’s presidency, but is it reasonable enough for people in Taiwan to bear such confidence in Mr. Trump’s anti-China sentiment? Or even the so-called Taiwan-friendly policies?

People can either proceed to follow suit behind the screen and make tons of wishful thinkings, which only will turn Taiwan into an ignorant country, or at least acknowledged the international circumstances before picking sides.

It’s more than a month before the president-elect Joe Biden take the office. Now it’s a good time to think about how constant relationships between the US and Taiwan can be established.
Operation KIPION explained: Protecting shipping lanes in the Gulf
By: Ministry Of Defence
Tags: Iraq, Military, Iran, Shipping, UK

Operation KIPION explained: Protecting shipping lanes in the Gulf

The Royal Navy has been present in the Gulf for more than 30 years securing the safe flow of trade goods, here are three things you need to know about the deployment Ministry of Defence Follow Dec 19, 2020 · 4 min read

Following the start of the Iran–Iraq War in September 1980, tensions were high in the region and a result the Gulf became an unstable and fragile area.

Since then, the Royal Navy and the Royal Fleet Auxiliary have been on patrol in the region 24 hours a day, seven days a week, 365 days a year.

Operation KIPION is the name given to the Royal Navy’s presence in the Gulf and Indian Ocean.

The protection that the Royal Navy and Royal Fleet Auxiliary give in the Gulf is pivotal. If the flow of ships and cargo in the region stops it will impact on everyone in the UK and more widely around the world.

Here are three things you need to know:

1. Why are we there?

In August 1980 through to 1988 a prolonged military conflict occurred between Iran and Iraq. The conflict at main was a large-scale ground-war, which resulted in many thousands of deaths on both sides.

But it was the naval aspect of that war which caused many of the issues that affect us to this day, in particular attacks on merchant shipping and oil tankers.

Many civilian seamen lost their lives and shipping vessels and oil tankers were damaged, causing large economic loss across the world.

The United Kingdom has enjoyed strong political, commercial and trading links in the Gulf region for decades, with 95% of Britain’s economic activity depending on the oceans.

Along with tension in the region, another legacy of the Iran–Iraq War is the large number of mines sown in and around the Gulf.

Promoting peace and security in the region helps to ensure the safe flow of oil and trade.
The CIA is running death squads in Afghanistan.
By: Manny Otiko
Tags: War, Foreign Policy, Afghanistan, Military, Trump

Photo by Daniel Stub on Unsplash

The CIA is running death squads in Afghanistan. And they’re targeting children. I wish I could say that’s surprising. We’ve seen this play before. And it usually doesn’t end well.
I’m not affiliated with the organization Veterans for Responsible Leadership, but I fully agree…
By: Matthew Spira
Tags: Morality, Current Events, Veterans, Military, Politics

I’m not affiliated with the organization Veterans for Responsible Leadership, but I fully agree with its goals as a combat veteran. As I read the recent New Yorker article about VFRL and its founder Dan Barkhuff, I was struck by this sentence: “Combat veterans learn that there are two kinds of courage — physical and moral.” This is 100% accurate in my experience. Being under fire, most soldiers do the right things. There are several ways we put it, but the bottom line is if there’s a bullet with your name on it, it’s time to go. What do you want to do, live forever? So you go out and execute. That’s physical courage.

On the other hand, moral courage is doing the right things at the right times over time. That can get tougher, especially when it isn’t always clear the right thing to do. And trying to make the right choices as you get worn down by the non-stop stress of the situation you’re in. Challenges to moral courage factor into moral injury; in turn, that ties into PTSD. But that’s a topic for another day.
Know These 5 Common Data Visualization Mistakes
By: William Chon
Tags: Data Visualization, Design, Programming, R, Data Science

Know These 5 Common Data Visualization Mistakes

Data visualization is one of the most powerful tools in analytics. It’s the best way to make sense of and communicate data to others. While a powerful tool, it is only as useful as the hands they are in. In visualizing data, as in any other field — there are rules, best practices, and guidelines. Of course, there are also mistakes.

I wanted to go over five of the most common mistakes that I’ve experienced and seen. I’ll focus on how these mistakes look in GGplot, but the universality of data visualization means that these lessons are applicable to other tools as well — Python, Excel, Tableau, etc.
Jets Fan Sentiment Analysis
By: Ryan Young
Tags: Text Analytics, R, Sports, New York Jets, Tableau

These results seem to incorrectly indicate that the sentiment was nearly evenly split between positive and negative emotions. There are several reasons for this. Many words are misinterpreted in this context, including the three words (in these dictionaries) that were used most frequently. “Win”, “lose” and “fans” are not the signals they traditionally may be. Jets fans are telling their team (or at least their social media person) what to do. In fact, the probability that any given “win” was preceded by a “don’t” is pretty high. This leads to another limitation…not using bigrams, as individual words often lose their context, especially when it comes to negation. It is also difficult to account for sarcasm, trolls and Jaguars fans, all of which likely contributed to the positive sentiments. Although, fans of the opposing team, who usually would also greatly distort the results, were not a factor in this case because Rams fans were very much not pleased with the game either.

To try and account for some of these limitations, I eliminated some of the most popular words that I believed were misinterpreted (such as “win”, “lose” and “fans”). I then used the “afinn” dictionary, which assigns a value for the sentiment of each word (ranging from -5 for the most negative to 5 for the most positive). I then summed the sentiment values of the tweeted words for each minute and created a plot.

This chart seems to accurately convey Jets fans’ sentiment, which negatively spiked following the touchdown that put New York ahead by three scores and then again when the Jets tweeted the final score at 7:01pm. This has to be one of the most unique fan sentiment charts you will ever see for a win.

If it’s any consolation to Jets fans, they will find a way to mess up their draft selection regardless of their position.

Merry Christmas, Jets fans. (And for those not celebrating, Happy March 300th.)
Using R and Random Forest to predict Diabetes Risk.
By: Spencer R Hall
Tags: Random Forest, Classification, Machine Learning, R, Data Science

Time for something a bit different from my previous posts, but hopefully more common in the future.

I was looking through the UCI Machine Learning Repository for a couple of data sets I could use for some simple machine learning problems to try interesting problems and keep my abilities sharp. This week I found the Early Stage Diabetes Risk Prediction. This comes from a paper on early prediction of diabetes risk. The abstract goes over the methods used, indicating Random Forest as doing particularly well. Since I don’t have access to the actual paper, I wanted to try out using R to try this model out.

It turns out, that this data isn’t terribly interesting when using Random Forest. Not much cleaning up or tuning is needed to get good accuracy. For the sake of completeness though, I’ll share my snippets and results.

First, we need to load the appropriate libraries and the data set. Here I load the columns except for age as factors, as well as clean up the column names.

library(tidyverse)

library(randomForest)

library(reprtree)

col_types = "dffffffffffffffff")

colnames(diabetesdf) <- make.names(colnames(diabetesdf))

names(diabetesdf)[names(diabetesdf) == 'class'] <- "result" diabetesdf https://archive.ics.uci.edu/ml/machine-learning-databases/00529/diabetes_data_upload.csv ",col_types = "dffffffffffffffff")colnames(diabetesdf)

Now, so we can test our out of sample error rate, let’s split 80% of the data for training our model and then test it with the remaining 20%

sample_size = round(nrow(diabetesdf)*.80)

index <- sample(seq_len(nrow(diabetesdf)), size = sample_size) train <- diabetesdf[index, ]

test <- diabetesdf[-index, ]

trainboost <- train

testboost <- test

trainboost$result <- as.numeric(trainboost$result) - 1

testboost$result <- as.numeric(testboost$result) - 1

Now, let’s train our Random Forest model using the default parameters and calculate the accuracy.

ddffit <- randomForest(result ~., data = train)

ddftest <- predict(ddffit, newdata = test, type = "response")

sum(ddftest == test$result)/nrow(test)

Getting an accuracy in prediction of around 96%.

Now, just for the sake of giving an interesting visualization, let’s use the reptree package by Abhijit Dasgupta. This enables us to create a visualization for the entirety of the Random Forest model made with the training set.

plot(ReprTree(ddffit, train, metric = 'd2'))

So, while this isn’t a particularly challenging data set for Random Forest to classify, it does provide an illustration on the basics of creating a model using Random Forest, as well as to try out reptree to make a visualization for the overall decision tree.

Since I know this data set works well and have more experience with implementing Machine Learning algorithms in R, I will come back to this to transfer more of my skill over to Python.
Machine Learning with R: A Complete Guide to Linear Regression
By: Dario Radečić
Tags: Machine Learning, Towards Data Science, Artificial Intelligence, R, Data Science

Multiple Linear Regression with R

You’ll use the Fish Market dataset to build your model. To start, the goal is to load in the dataset and check if some of the assumptions hold. Normal distribution and outlier assumptions can be checked with boxplots.

The code snippet below loads in the dataset and visualizes box plots for every feature (not the target):

Image 6 — Boxplots of the input features (image by author)

A degree of skew seems to be present in all input variables, and the first three contain a couple of outliers. We’ll keep this article strictly machine learning based, so we won’t do any data preparation and cleaning.

Train/test split is the obvious next step once you’re done with preparation. The caTools package is the perfect candidate for the job.

You can train the model on the training set after the split. R has the lm function built-in, and it is used to train linear models. Inside the lm function, you’ll need to write the target variable on the left and input features on the right, separated by the ~ sign. If you put a dot instead of feature names, it means you want to train the model on all features.

After the model is trained, you can call the summary() function to see how well it performed on the training set. Here’s a code snippet for everything discussed so far:

Image 7 — Summary statistics of a multiple linear regression model (image by author)

The most interesting thing here is the P-values, displayed in the Pr(>|t|) column. Those values indicate the probability of a variable not being important for prediction. It’s common to use a 5% significance threshold, so if a P-value is 0.05 or below, we can say that there’s a low chance it is not significant for the analysis.

Let’s make a residuals plot next. As a general rule, if a histogram of residuals looks normally distributed, the linear model is as good as it can be. If not, it means you can improve it somehow. Here’s the code for visualizing residuals:

Image 8 — Residuals plot of a multiple linear regression model (image by author)

As you can see, there’s a bit of skew present due to a large error on the far right.

And now it’s time to make predictions on the test set. You can use the predict() function to apply the model to the test set. As an additional step, you can combine actual values and predictions into a single data frame, just so the evaluation becomes easier. Here’s how:

Image 9 — Dataset comparing actual values and predictions for the test set (image by author)

If you want a more concrete way of evaluating your regression models, look no further than RMSE (Root Mean Squared Error). This metric will inform you how wrong your model is on average. In this case, it reports back the average number of weight units the model is wrong:
Is there Gender Inequality in the workplace?
By: Hui An Huang
Tags: Practice, R, Gender Pay Gap, Sociology

College degree’s best model:

In the previous models created by job titles, the pay increases with the higher Age and Seniority in the data. However, in the model, the pay of male is higher than female, but the average Age and Seniority are:

Discussion

The gender ratio of most job titles in this database is 1:1, and there are only a few occupations that are exceptions. It does not seem to fit the real social situation. Previous studies have shown that female labor participation has made vital progress in the past half-century, but employment segregation by gender and gender pay gap still exists. Although these differences are shrinking with social progress, there is still more room for improvements. (World Bank (2011))

However, several trends can still be seen from this data. Calculating the average annual pay of all respondents according to gender shows that the average pay of male is slightly higher than that of female. Further analysis can indeed reveal that the average pay of male and female is higher and lower between different job titles. Still, the enormous pay gap between male and female is 11886.43 USD by Software Engineer, and male are paid higher than female.

In the two Total Pay models, the factors that significantly affect the annual pay are Education, Age and Seniority. The study further analyzes the average years of education by gender. It can be found that male received longer education than female. Judging from the number of people in different education levels, male and female are almost the same before college, but more male had master’s and doctoral degrees.

Looking closely at the gender pay gap based on education levels, it can find that female’s annual pay in every section is lower than male. There is the most significant difference in bachelor respondents, which even reaches 11755.01 USD.

Generally speaking, although “Gender Inequality” is not obvious in this dataset, it is still visible in the analysis. It is satisfying to know that many things have improved with the progress of society over the past few decades, but inequality is still hidden in many aspects and needs to be resolved with the promotion of regulations and awareness.

Code Appendix

＊＊＊Please visit the following link to access to the R code＊＊＊

https://docs.google.com/document/d/1xxwtVBLIfvle7HEijmwDe66nJPxAmlbpFuL4f294dJI/edit?usp=sharing

References

World Bank. 2011. “Gender Differences in Employment and Why They Matter.” World Development Report 2012: Gender Equality and Development, no. 5.

Jacobs, Jerry A. 1996. “Gender Inequality and Higher Education.” Annual Review of Sociology 22 (1): 153–85.

Treanor, Jill. 2016. “Gender Pay Gap Could Take 170 Years to Close, Says World Economic Forum.” The Guardian.

Zakharchuk, Olha. 2020. “The Gender Pay Gap: Main Causes and Features in Different Countries of the World.”

Zou, Min. 2015. “Gender, Work Orientations and Job Satisfaction.” Work, Employment and Society 29 (1): 3–22.
100 ggplot2 tips
By: Kamil Szymkowski
Tags: Data Visualization, Ggplot2, Ggplot, Tips, R

A collection of useful information on the package ggplot2 as well as on the underlying philosophy known as “The Grammar of Graphics”.

GitHub repository available here.

💡 1. ggvis = interactive ggplot2

The ggvis package is a successor to ggplot2, which has been created with the goal of extending the static ggplot2 to web and interactive graphics.

💡 2. Facetting

You can use the facet_wrap(~variable) to produce the same plot for every factor level of the supplied variable — if it is categorical, or for every quantity if it is continuous.

ggplot(mpg, aes(cty, hwy)) +

geom_line() +

facet_wrap(~manufacturer)

💡 3. geom_histogram() vs geom_bar()

The geom_histogram() is used to show the distribution of continuous variables, whereas the geom_bar() is used to show the distribution of categorical variables.

ggplot(mpg, aes(cty)) +

geom_histogram()

ggplot(mpg, aes(manufacturer)) +

geom_bar()

While the geom_bar() will work with both continuous and categorical variables, the geom_histogram() will return an error message if it's supplied a categorical variable.

💡 4. geom_line() vs geom_path()

The difference between geom_line() and geom_path() is shown below:

It all comes down to the variable x. When the variable x is ordered in the supplied data set, the two geoms will produce the same plots. However, when the variable x is not ordered, then the two geoms display different behaviour:

geom_line() will first put x in order and only then will draw lines

will first put x in order and only then will draw lines geom_path() will draw lines between points as they appear in the data

This behaviour of the geom_path() makes it possible to draw any shape:

💡 5. Save a plot with ggsave()

The function ggsave() can be used to save the most recently created plot to a specified file path.

Example 1. Saving the plot with name “p” as PNG to the current directory:

ggplot(no_order, aes(x = varx, y = vary)) +

geom_path() ggsave("p.png")

Example 2. Saving the plot to a folder “grammar-of-graphics”:

ggplot(no_order, aes(x = varx, y = vary)) +

geom_path() ggsave("./grammar-of-graphics/p.png")

💡 6. geom_area() vs geom_polygon()

The geom_area() draws area plots, while the geom_polygon() draws filled paths.

The latter can therefore be used to draw any filled shape:

💡 7. Use geom_label() instead of geom_text()
How to get Bitcoin futures market data from Deribit using R
Tags: Trading, Crytocurrency, R, Data Science, Bitcoin

Easy step-by-step instructions on getting market data from Deribit into R

This brief note this to help Bitcoin futures trades extract live data from Deribit onto their own R programming environment so that they can conduct quantitative trading analysis.

Photo by Pierre Borthiry on Unsplash

Firstly, the R package we will be using to help us extract / log live data from Deribit is called rderbit . You can find its code repository below:

GitHub — paltsev-p/rderibit: Deribit API interface for R

As of writing, this package works very well. The only problem I’ve detected is that it uses V1 of the Deribit API. Deribit has since released V2, so at some stage this library needs to be updated. So far, I am still able to extract market data using this R package.

Step 1. Download rderibit R package

We use devtools because the package is not available on CRAN

devtools::install_git("https://github.com/paltsev-p/rderibit")

Step 2. Load package library

library(rderibit)

Step 3. Scan live bitcoin market

instrument <- "BTC-PERPETUAL"

scan <- getsummary(instrument)

Attributes inside object Scan$result

The object scan should give you a high level snapshot of the Bitcoin futures market. It will give you i) funding ii) open interest, iii) daily high low and volume, iv) current bid-ask spreads and v) current price along with a timestamp.

For instance,

Step 4. Get orderbook depth data

instrument <- "BTC-PERPETUAL"

tmp <- getorderbook(instrument, simplify = TRUE)

For instance,

Step 5. Get last 1mins worth of trades

mins <- 1

currTime <- as.numeric(as.POSIXct(Sys.time())) * 1000

startTime <- currTime - mins * 60 * 1000

trades <- getlasttrades(instrument, count = 100, startTimestamp = startTime, endTimestamp = currTime)

Here we use 1 minute, but you can easily change this to something else

Hope this helps!
Explaining predictions with triplot, part 2
By: Katarzyna Pękala
Tags: Explainable Ai, Machine Learning, R, Xai

We are going to observe how local triplot and predict_aspects work by using them on the same dataset that was explored in the previous blog post — FIFA 20 (dataset description) from Kaggle.

library("DALEX")

data(fifa)

fifa$value_eur <- fifa$value_eur/10⁶

fifa[, c("nationality", "overall", "potential",

"wage_eur")] <- NULL

In the beginning, we get the FIFA dataset from the DALEX. There are a couple of variables that could be treated as target (player’s potential, wage, etc). Among them we choose player’s value in Euro as the predicted variable and remove the rest of them, as well as we remove nationality variable. We also rescale the target, so it’s more readable.

After the data preparation step is finished, we build a random forest model from ranger package. Afterward, we wrap it in a DALEX explainer.

library("ranger")

set.seed(2020)

fifa_model_rf <- ranger(value_eur~., data = fifa)

fifa_explainer_rf <- DALEX::explain(fifa_model_rf,

data = fifa[,-1],

y = fifa$value_eur,

label = "Random Forest")

Main difference between global and local triplot in terms of usage, is providing local triplot with the selected observation. In this example, we are looking to explain the model’s prediction for the the player with the highest value in Euro.

top_player <- fifa[order(fifa$value_eur, decreasing = TRUE),][1,]

Now, we can finally call predict_triplot. We provide it with some additional parameters: we are using Pearson correlation method and setting the sample size to be at 5000.

fifa_triplot_local <- predict_triplot(fifa_explainer, top_player,

N = 5000,

cor_method = "pearson")

plot(fifa_triplot_local)

In result we see local triplot, a picture that is an analog of the global triplot.

Local triplot for top_player

Here we can see in one image:
Introduction to Data Science with R
By: Zidane Aldani Fitrah Ramadhan
Tags: Data Science, R, Data Enthusiast

Introduction to Data Science with R

R is the name of a programming language as well as software for processing data and graphics.

R is very popular today because of three things, namely:

Many data processing options with a very complete number of features.

It is faster to learn and run to process data compared to other languages.

R is free and open source, which means there is no need for licensing fees which are usually very expensive for data processing software .

R is so popular also because it is used by the world’s big companies.

Some of them are AirBnB for data science, Uber for statistical analysis, Facebook for behavior analysis, and others.

R “Hello World”

Input

Output

Calculation of Numbers with R

Input

Output

Note:

Where 15 is the result of 10 + 5.

Variable in R

The use of variables in programming is very important to accommodate numbers and text in R with a name.

Input

Ouput

Note:

This means that variable a is filled with the number 5.

print() used to print the contents of the variable a

Comment on R

Comment is text to add information to our code, so we'll remember what we did when we opened the code again.

Comments are not considered executable code.

In R the use of comments is to precede a text with a '#' sign.

Input

Output

From this process it can be seen that the comments are not processed by R, so only the mathematical calculation results are displayed.

Vector on R

Vector is a data structure that can store more than one data used in R.

Its use is very simple, namely using the c function along with the data you want to store.

Input

Output

You can see that the c command (5, 10, 15) creates three series of numbers, namely 5, 10 and 15 which are stored and displayed together in a vector.

Input

Output

You can see that the c command (1:10) creates a vector with ten series of numbers starting with 1 and ending with the value 10.

Using the Summary Function

R is an analysis function called summary that can be used to deduce the data that we are processing again.

Input

Output

This means that from the vector there is a minimum number 1 (Min), a maximum number 10(Max), an average number 5.50(Mean), and a middle number 5.50(Median). For 1st Qu and 3rd Qu we just ignore it.
Visualizing regression point-estimates in R
By: Lara Southard
Tags: Statistics, Data Visualization, R, Data Science, Regression

You can run your regression analysis (any variation) and then apply the following code:

model.glm = glm(as.numeric(outcome == "Y") ~ Group + var1 + var2 +

Group*var2,

#specific to your model,

family= gaussian(link="identity"),

data = model.df) results <- as.data.frame(get_model_data(model.glm, type = "eff", terms = c("Group"))) %>%

mutate(Group = if_else(x == 1, "Group 1", "Group 2"),

predicted = (predicted*100),

conf.low = (conf.low*100),

conf.high = (conf.high*100)) %>%

dplyr::select("Group", "pp" = "predicted", "std.error",

"conf.low", "conf.high")

Review the different type = … but type = "eff" means that you don’t have to use a true reference group, but instead, you can use a proportion of your data. My goal in the code above was to get predictive probabilities which allows a simple way to explain regression effects. get_model_data() comes from library( ggeffcts() ). I like this method when I’m using ethnicity or any other time where there’s potential for the results to be biased.

Then you can visualize it using this code:

ggplot(results, aes(x= Group, y = pp))+

geom_errorbar(aes(ymin=conf.low, ymax=conf.high), width=.1) +

geom_point(size = 3)+

labs(y = "% Predicted Probability", x = "Group")+

scale_y_continuous(limits = c(0, 100), expand = c(0,0))+ #in case you want the customization

theme(axis.line = element_line(color = "black"),

axis.title.y = element_text(size =16),

axis.title.x = element_text(size= 16),

axis.text.x = element_text(color = "black", size = 16),

axis.text.y = element_text(size = 16, color = "black"),

panel.background = element_blank(),

panel.grid.major=element_line("gray"),

panel.grid.minor=element_line("gray"),

text=element_text(size = 14, family = "serif")

)

Here is what the code above will produce:

The benefit of visualizing regression coefficients this way is simplicity. We can explain that if the error bars overlap then the effect is nonsignificant, but since they don’t here, it is significant. Also, the outcome is higher for the dot that’s higher up on the plot. This particular example leverages predicted probabilities, but there are other options in the code.
Great post! you could have a look to **reprex** addins to build reproducible examples.
By: Josep Maria Porrà
Tags: Reprex, R

Great post! you could have a look to **reprex** addins to build reproducible examples.
Calendar Heat Maps in R with tidyverse
By: Maria Ma
Tags: Tidyverse, R, Data Science, Heatmap

First thing is to fill in the rest of the relevant dates that we have no data for. This is done with complete() which is a really neat tidyr function. It’ll generate a sequence of dates from an input. Here, I want dates from September 1st to the end of the year.

data %>%

complete(date = seq.Date(as.Date("2020-09-01"), as.Date("2020-12-31"), by="day")) %>%

mutate(month = month(date, label = TRUE),

wday = wday(date, label = TRUE),

day = day(date),

week = epiweek(date)) -> df
[Win10]R語言－Tensorflow環境設定
Tags: R, TensorFlow

紀錄在windows上利用R來使用Tensorflow

１．安裝Anaconda

直接到官網下載，安裝即可，中間勾選將Anaconda加到path中。

２．在Anaconda中創造一個r-tensorflow的虛擬環境

R的keras套件，會去找名為r-tensorflow的虛擬環境來使用，故名稱需固定。

終端機： conda create --name r-tensorflow

３．進入r-tensorflow虛擬環境中，安裝設定tensorflow

使用環境

activate r-tensorflow

安裝 r-tensorflow keras

(r-tensorflow) C:\> pip install --ignore-installed --upgrade tensorflow

(r-tensorflow) C:\> conda install -c conda-forge keras

４．測試環境內安裝是否成功

C:\> activate r-tensorflow

(r-tensorflow) C:\> python # Python codes for tensorflow

>>> import tensorflow as tf

>>> hello = tf.constant('Hello, TensorFlow!')

>>> sess = tf.Session()

>>> print(sess.run(hello))

５．R安裝相關套件

R

install.package("tensorflow")

install.package("keras")

６．匯入套件並測試
Missing Values in R — remove na values
By: Kayren
Tags: Missing Data, Statistical Analysis, Missing Values, R, Data Science

The first method — is.na()

is.na tests the presence of missing values or null values in a data set. The method searches through every single column of the dataset, finding outliers with a na value that might affect the calculation.

Example;``` x <- c(1,2,3,4,NA) is.na(x) returns a series of FALSE and TRUE depending on whether the values of the vector have na values. The output in this case would be FALSE FALSE FALSE FALSE TRUE```

Second method — na.omit()

Here’s a sample dataset with missing values.

a dataset with missing values. Screenshot from R studio.

na.omit() method removes the rows with na values from a list. The na.omit() function returns a list without any rows that contain na values. This is the faster way to remove na values in R.

Complete cases complete.cases() — Returns vector of rows with na values.

The na.omit() function relies on the sweeping assumption that the dropped na rows are similar to the typical member of the dataset, and are not total outliers whereas the complete.cases() allows you to perform a more detailed review and expression.

Removing the na rows in a dataset might not be the right decision here and we might therefore consider inspecting datasets of the original data to evaluate if other factors are at work.

We accomplish this with the complete.cases() function. This R function will examine a data frame and return a result vector of the rows which contain missing values. We can examine the dropped records and purge them if we wish.

Missing values removed using the complete.cases() function.

A display of the removed missing values which can be examined.

Fix in place using the na.rm

Another way of dealing with missing values is by using the na.rm logical parameter. When na.rm is True, it skips over the na values. However, when the na.rm is False, then it returns NA from the calculation being done on the entire row or column.

The rows with the value of na are retained in the data frame but excluded in relevant calculations. This is often the best method if you find that there are significant trends in the observation, with na values. Support for this varies by package so please check the documentation for your specific package.

notice the na value on the rowSums because we had an na value in the dataset on column 2 row 2

Dealing with missing data from a dataset is critical to proper data science. R makes dealing with this missing data so easy that's why it is often used in statistical analysis.
Generating Random Numbers In R
By: My Master Designer
Tags: Machine Learning, Artificial Intelligence, Programming, R, Data Science

In this first simulation tutorial, we going to look at generating random numbers for simulation in r programming. If you read the previous articles you can see simple functions to generate random numbers. Let’s begin with to first subject.

If you are new to r programming and do not know certain topics and want to practice, you can improve yourself by starting here. From here, you can review data visualization topics and improve yourself with exercises.

What Is Simulation And Why We Do Simulation?

simulations are experiments created to try out certain projects. If you do not have data for an experiment, you can create your own data with random numbers. It is also used to check the output. You can experiment with simulation to find errors in your project.

Rnorm Function Syntax

It is used to generate random numbers with mean and standard deviation with the rnorm function. where the average of all random numbers generated is equal to the value used in the function.

rnorm(n,r)

n = number of data to be generated

r = the ratio of randomly generated numbers

Rnorm Function Example

Now let’s see what we have learned in a real example. In this example, we will construct 10 random numbers after that we going to use these random numbers for creating graphics. If you don’t know creating a graphic you should read this article.

r$> x <- rnorm(10 , 1)

r$> y <- rnorm(10 , 2)

r$> plot(x , y)

Continue In My Master Designer.
Animated bar race ranking in R
By: Kamil Szymkowski
Tags: GIF, Data Visualization, Ggplot2, Gganimate, R

Instructions with code for how to create a bar race ranking GIF animation in R.

GitHub repository is available here.

Step 1. Packages

The packages below need to be installed prior to running the later code:

library(ggplot2)

library(gganimate)

library(tweenr

library(magick)

library(gifski)

library(dplyr)

library(magrittr)

library(countrycode)

library(grid)

Our ggplot will include flags positioned on the vertical axis. The package enabling this is called “ggflags”, however it is not available on CRAN. That’s why one needs to install it using devtools, as shown in the code below. For the installation to be successful, it is necessary to install devtools first, which in turn requires Rtools to be present in the operating system.

devtools::install_github("ellisp/ggflags")

library(ggflags)

Step 2. Data

The data set represents total exports from Scotland to other countries for different years. Although the excel spreadsheet below includes multiple columns, for the sake of this exercise important are only columns: “Year”, “Country”, “Total” and “Rank”. The data is available here.

In the data-manipulating code below, accomplished are three things:

Top 25 countries for each year are kept, other entries are discarded. The variable ‘Rank’ is multiplied by 120, this is to scale bars. A new column representing ISO country codes is added.

# DATA

COUNTRIES <- read.csv("COUNTRIES.csv")

D <- COUNTRIES %>%

group_by(Year) %>%

filter() %>%

top_n(n = 25, wt = Total) %>%

mutate(Rank = Rank * 120) %>%

ungroup()

D <- D[with(D, order(Year, -Total)),] D$Code <- tolower(countrycode(D$Country, "country.name", "iso2c"))

As a result of the data manipulation, the new table takes the following form:

Step 3. Plot

Next comes the creation of the ggplot to be animated. Please refer to inline comments to view explanations for each line of code.

Step 4. Animation

The plot above has been assigned to the variable “p”. This is then fed to the function animate() from the package “gganimate”. It also requires that the user specifies the frames per seconds, duration of the animation , and the size in pixels.

# ANIMATION

animate(p, fps = 30, duration = 30, width = 800, height = 600)

This will then render the animation.

Voila!

GitHub repository is available here.
Maternal Mortality in New Jersey: An Interactive Social Determinant of Health Tool
By: Doug Friedman
Tags: Data Science, Open Source, Womens Health, R, Maternal Health

While the COVID-19 pandemic has dominated every aspect of our lives these past few months, another public health crisis has been burning in the background. The U.S. is in the middle of a maternal mortality crisis. The maternal mortality rate in the U.S. is the highest of all developed nations and is rising.[i] A woman today is twice as likely to die in childbirth than her mother was a generation ago.[ii] Healthcare professionals, public health officials, and policy makers are all working to understand the drivers of maternal mortality. In addition, there are stark racial disparities. Black women die during pregnancy and childbirth at a rate 3–4 times above the rate of white women.[iii] The root cause for this difference is not well understood, but healthcare inequities, structural racism, social determinants of health and bias all play a role. Combined with recent events highlighting structural racism in the U.S., this crisis has taken on new urgency.

According to the CDC, “700 women die each year in the United States as a result of pregnancy or delivery complications.” [iv] This figure has been on the rise overall since the Pregnancy Mortality Surveillance System was implemented in 1986.[v] However, these figures have been notoriously difficult to calculate as pregnancy-related mortality data has been historically poor and/or inconsistent.[vi]
How to Make Stunning Line Charts in R: A Complete Guide with ggplot2
By: Dario Radečić
Tags: Data Visualization, Ggplot2, Towards Data Science, R, Data Science

Make your first line chart

R has a gapminder package you can download. It contains data on life expectancy, population, and GDP between 1952 and 2007. It’s a time-series dataset, which is excellent for line-based visualizations.

Here’s how to load it (and other libraries):

library(dplyr)

library(ggplot2)

library(gapminder)



head(gapminder)

Calling the head() function outputs the first six rows of the dataset. Here’s how they look:

Image 1 — Head of Gapminder dataset (image by author)

R’s widely used package for data visualization is ggplot2 . It’s based on the layering principle. The first layer represents the data, and after that comes a visualization layer (or layers). These two are mandatory for any chart type, and line charts are no exception. You’ll learn how to add additional layers later.

Your first chart will show the population over time for the United States. Columns year and pop are placed on X-axis and Y-axis, respectively:

usa <- gapminder %>%

filter(continent == "Americas", country == "United States")



ggplot(usa, aes(x = year, y = pop)) +

geom_line()

Here’s the corresponding visualization:

Image 2 — Population growth over time in the United States (image by author)

The visualization is informative but as ugly as they come. The following sections will show you how to tweak the visuals.
coRona data #3: automatizace na Google cloud platform
By: Jiri Stepan
Tags: Data Studio, Covid 19, R, Google Cloud Run, Google Cloud Platform

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Function In R Programming
By: My Master Designer
Tags: Data Analysis, R Programming, Programming, R, Data Science

In this tutorial, we going to look at functions in r programming also going to learn arguments, this tutorial is to continue of the previous article. Here, you can access the previous article.

What Is Function And Why We Use Function?

functions are blocks of code that execute a basic piece of code without rewriting it. For example, you can always type a + b to sum, or you can easily get it done with the addition function.

Functions are the structures that organize the code. They make the code more readable and facilitate continuous work. They will constantly avoid the hassle of copy and paste because you can use it to call your code blocks.

Function Syntax In R

functionName <- function(arguments)

{

#Code

}

In this syntax, the name of the function is written first, then the function keyword and arguments are added. Let’s examine a simple function without arguments then we going to create a new function with arguments.

HelloWorld <- function()

{

print("Hello World!")

}

HelloWorld()

After creating the function, you can use the name and parentheses to call it just like any other function. This function will only print hello world on the screen when it is called, but in the following stages, we will do more specific tasks such as printing the values entered by the user on the screen.

Write <- function(message)

{

message

}

Write("Hello World!")

Continue To My Master Designer
What is Tidy Data?
By: Benedict Neo
Tags: Data Analysis, Tidyverse, Machine Learning, R, Data Science

Data Cleaning & Tidying

What is Tidy Data?

Photo by Hunter Harritt on Unsplash

Introduction

There’s a popular saying in Data Science that goes like this — “Data Scientists spend up to 80% of the time on data cleaning and 20 percent of their time on actual data analysis”. The origin of this quote goes back to 2003, in Dasu and Johnson’s book, Exploratory Data Mining and Data Cleaning, and it still true to this day.

In a typical Data Science project, from importing your data to communicating your results, tidying your data is a crucial aspect in making your workflow more productive and efficient.

The process of tidying data would thus create what’s known as tidy data, which is an ideal first formulated by Hadley Wickham in his paper. So my article will be largely a summarization or extracting the essence of the paper if you will.

What is Tidy Data?

From the paper, the definition given is:

Tidy datasets provide a standardized way to link the structure of a dataset (its physical layout) with its semantics (its meaning)

To break down this definition, you have to first understand what structure and semantics means.

> Structure is the form and shape of your data. In statistics, most datasets are rectangular data tables(data frames) and are made up of rows and columns.

> Semantics is the meaning for the dataset. Datasets are a collection of values, either quantitative or qualitative. These values are organized in 2 ways — variable & observation.

Variables — all values that measure the same underlying attribute across units

Observations — all values measured on the same unit across attributes

If you didn’t get any of that, I recommend you reading the paper mentioned as it has examples and tables that illustrate it better.

Nonetheless, the 3 rules of tidy data help simplify the concept and make it more intuitive.
Best Smartphones in Rs 10000
By: Reema Sharma
Tags: Best, Blog, Blogging, Rs, Smartphones

Best Smartphones with 64GB Storage for Rs 10000

Moto E7 Plus: The phone costs Rs 9,499. The phone has a 6.5-inch HD Plus display, Snapdragon 460 processor, 4GB RAM…
Manipulação de dados: uma introdução à coleção de pacotes Tidyverse no R
By: Amanda Munari Guimarães
Tags: Data Visualization, Statistics, Data, R, Data Science

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Dashboard in PowerBI: II — Loading a complex CSV file with an R script
By: Jan Hapala
Tags: Power Bi, R

In this article, we will load a sales table into Power BI Desktop. Unfortunately, the file that we have been given by the client is not a simple table with one header and one body. Instead, the file contains multiple tables concatenated one below another. At least, these tables follow the same format. But how do we import such a table into PowerBI?

Use R

To make things more complicated, we might expect that the number of subtables may change in the future (e.g. when new work groups are established) and our solution should be robust enough to handle such scenario.

We will not get by here by the mere power of Power BI and we will have to call in higher powers. In such situations, one of the script (Python or R) feature will solve our problem. I prefer using R for working with tabular data, thus I will present a solution integrating an R script in the Power BI Desktop.

Step by step

1. Review the table

First, let us review the file in a spreadsheet and see its format.

a. The input CSV file: a table composed of multiple tables in the same format.

The table is composed of multiple tables in the same format. Each of the sub-tables holds sales data for one work group (see the cells B1 and B161).

We want to load this data, cut it into subtables and combine them back into one table with one header and a column specifying the respective work group.

2. Develop the R script outside Power BI

Here comes the point when a bit scripting experience is handy. First, I write a script and test it in RStudio.

b. The R script shown in the RStudio

The script breaks the table down into multiple tables based on the separation rows (those containing the word “Workgroup” in the first field) and combines the subtables into one table with the rbindlist() function.

3. Copy the R code into Power BI

We need to add New Source → R Script in the Power Query Editor and paste our R code into the window. When we then open the Advanced Editor, we will see our code collapsed into one line:

c. R script in the Advanced Editor

4. Optional: Change the data file path

As shown in the part I of this Power BI tutorial, we can change the absolute path to our input data into a dynamic one (e.g. when we need to swap our real data with randomly generated). We can do this either in the Advanced Editor or in the Applied Steps (in the pictures below).

d. Absolute path to our CSV file

e. Dynamic path to the CSV file

Notice the three quotation marks before calling the function PickSourcePath(). Two quotation marks are an escape sequence for a quotation mark in the M language, the third one closes the string.

Summary

In this article, we have seen how to import dynamically a complex CSV file into Power BI Desktop with the use of an R script.

The solution presented in this article can be downloaded from GitHub.
Loop Functions In R Programming
By: My Master Designer
Tags: Code, Data Visualization, Programming, R, Data Science

In this tutorial, we going to learn the loop function, in another saying we going to learn apply,lapply,tapply, and mapply if you don’t know data types, you can read Data Types In R.

What Are Loop Functions?

You can operate on data types with the loop functions, these functions save you from creating a loop over and over again. It is also possible to apply the operations you want to all data with loop functions.

Apply Loop Function

The apply function is used to operate on a matrix or data frame, resulting in a vector, list, or array. The syntax is simple, the first argument is matrix or data frame, the second argument is used to select a column or row, and the last argument specifies the action to be taken.

x <- matrix(1:6 , ncol = 3 , nrow = 2)

apply(x , 1 , sum)

In here, we have gathered up the rows of the matrix. If we are going to operate on the columns, it is sufficient to write 2 instead of 1.

Lapply Loop Function

The lapply function is used to operate on a list, the output is the same length as the input. Each element is the result of applying the operation to the corresponding element of the list.

x <- list(c(1 , 2 , 3) , c(4 , 5 , 6) , c(7 , 8 , 9)

lapply(x , mean)

very similar to the apply syntax, a list is given to the first argument and 2nd argument the action to be taken on the is written. In the example here, each index of x will be averaged and written to the same index on the output list.

Continue Read My Master Designer.