fabiochiu/medium-articles huggingface dataset:
How to Use Mysql Password Encryption with NodeJS, Express and Bcrypt
Tags: React, MySQL, JavaScript, Bcrypt, Expressjs

Dave’s not here, man

Note: this article begins after you’ve setup and coded an Express Router on a NodeJS server and written an API that connects to a mysql database

Now that we’ve got our sexy app up and running, we want to make it a little more sophisticated. We didn’t bother to include user-password encryption before, because who cares about hacking my demo app for school, right?

Everrryyyyyybbboooodddddddyyyyyy!!!!

We’re going to be using a package called bcrypt to encrypt our password and for the validation. Here’s what wikipedia has to say about bcrypt:

bcrypt is a password-hashing function designed by Niels Provos and David Mazières, based on the Blowfish cipher and presented at USENIX in 1999.[1] Besides incorporating a salt to protect against rainbow table attacks, bcrypt is an adaptive function: over time, the iteration count can be increased to make it slower, so it remains resistant to brute-force search attacks even with increasing computation power.

So it’s been around a lot longer than you might figure, but the algorithm is still available to you in React by including the bcrypt package in your project:

npm install bcrypt

Now in our controllers, where we keep the sql queries for the API, we’re going to edit whatever file our login functions exist in and add the following at the top:

const bcrypt = require('bcrypt');

const saltRounds = 10;

A saltRound is

With “salt round” they actually mean the cost factor. The cost factor controls how much time is needed to calculate a single BCrypt hash. The higher the cost factor, the more hashing rounds are done. Increasing the cost factor by 1 doubles the necessary time. The more time is necessary, the more difficult is brute-forcing.

So we chose 10, which is a pretty big number for a small app, but it’s also not big enough to cause any noticeable lag. Play around with it if you want, but unless you’re building an enterprise app with millions of users, you don’t have to go much further.

Important: This implementation will not affect any passwords that are already stored in the database as plain text. In fact, any passwords that already exist will not work any more!

One thing you may have to do is to change the datatype of the password column in your mysql table. For instance, if you just set it up as char(25) to hold a text password, you may have to alter it to hold the hashed password variable.

Here’s an excellent resource to remind you how to alter a column’s datatype in a mysql table if you need it.
How to mitigate cache storming in a high traffic
By: Ebrahim Poursadeqi
Tags: MySQL, Cache, Database, Memcached, Dogpile

Problem

In our recent promotions, we were facing a strange behavior in our database. A normal load on the database turned to a huge load all of a sudden with repetitive queries on a database.

Some of the queries were expensive to generate so we tried to improve queries. However, this didn’t solve all of the cases. Although these queries were on a cache block, there were a lot of queries like that on the database. So what was the problem?

Root Cause

Actually, the problem was that we were in the promotion and we had a lot of concurrent users, who wanted to open the same page, and if cache service returned a miss for a key, a lot of users requesting that data from the database. This behavior is called cache storming, or dog-piling.

In order not to face cache miss in production, we tried to prefetch data before missing. We used to have a function like this to get the data.

public function getData(String $key, Closure $closure){

$cacheValue = $this->getFromCache($key); if(!$cacheValue){

$cacheValue = $closure()

$this->setCacheValue($key,$cacheValue);

} return $cacheValue;

}

Solution:

We tried to use a simple formula in order to check whether we should prefetch the value of a cache key or not.

shouldPrefetch = now — cache update time > time to fill cache + a fixed threshold

time to fill can control on not prefetching a cache key too early, and a fixed threshold can have control on not prefetching the cache key too late.

Then, we changed the getData to

public Result getData(String key, Closure closure, bool forceUpdate = false){

cacheValue = this.getFromCache(key);

if(!cacheValue || forceUpdate){

startTime = now;

cacheValue = closure()

timeToFill = now - startTime this.setCacheValue(key,cacheValue);

this.setCacheValue(this.getMetaKey(key),[now,timeToFill]);

}

return cacheValue;

}

We didn’t save cache key meta date in the same key. Rather, we stored it in another cache key generating by getMetaKey so that we have a much lightweight response when we are checking whether we should prefetch or not.
Easy Implementation of Transactions Handling in MySql With Spring Boot…
By: Yasindu Wishmith Korawage
Tags: MySQL, Mapping, Datasource, Yml, Transactions

In this short blog, I have exposed an easy code level implementation of transaction handling.

I have used two commands for handling transaction.

1. COMMIT=> To save changes 2. ROLLBACK=> To roll back changes

For this implementation, I have used the DataSource factory. It can be configured with any physical data source. There are a few methods to configure DataSource in spring boot. I have adduced two methods as followings.

1. Create a class for configuration.

2. Configuration with Application.yml/properties file

Usually, In spring boot project, we configure a database in the application.yml or application.properties without above code implementation as below.

To handling transaction, we want a Connection with the database. If you configure DataSource with a class, you can get a connection easy as following code line.

However, If you have used YAML or properties file, you would create another class for configuring DataSource with mapping configurations from YAML or properties file as follows.

Then you can establish the connection with the following code line.
Free SQL Public Trello Resource Board
By: Lucas Solmes
Tags: MySQL, Relational Databases, Sql

Check out the link & follow along, click here
Build AI Powerd Chat Bot with php
By: Avishka Dev
Tags: MySQL, PHP

How to Build Chatbot using php and MySQL

Avishka Dev |My Project |About Me | Linkedin | GitHub | Twitter | Facebook
Best way to migrate from EC2 mysql to RDS — e.g 2.5Tb in 12 hours
By: Sourabh Kalal
Tags: MySQL, Dumps, Restore, Mysqldump, Rds

I had been looking for a solution where I can migrate Tbs of data from EC2 -mysql to RDS. and I came up with a way which was obvious tool “mysqldump”. I have tried taking dump and restore, and it took 20+ hours for just taking the dump and another 20+ hours to restore.

So, it’s like 40+ hours for dumps and then start replication where we need to wait for replication data of last 2 days due to high amount of time taking in dump.

and I have decide to use mydumper which was taking just 10–12 hours for the same work.

We can do the same thing either with replication — Where we need to take the dump, restore and start replication with source, and once they are with same data then stop slave and switch the URLs of db. or Take some downtime for write operations and take the dump-restore and switch the domain.

I’ve decided to take some downtime(10h) for write operations, and use the mydumper with some tricks and tweaks.

PREREQUISITE:

Create a domain name to point to your database ip and make the changes in your application to start using that domain. e.g mydb.something → 10.0.0.1

Create a RDS and with same mysql version and parameter group. Disable encryption, backup, multi-Az, enhanced monitoring everything.

Make the below changes in parameter group:

Set innodb-flush-log-at-trx-commit = 0

Install mysql to your local, it will help while restoring mysql users.

Install mydumper utility in source (EC2) instance:
CRUD Operation In PHP
Tags: MySQL, Operation, HTML, PHP, Crud

Hello All,

In this tutorial I will show you how to create CRUD operation in PHP. So if you are newcomer in developing or fresher in web developing then this post definitely help you to learn how to create login-logout with insert, update, delete data using MySQL in database.

So, let’s start and follow my steps as described in below and get output.

Step 1 : Create Folder And Add New File

Step 2 : Create Signup Page

Step 3 : Create Login Page

Step 4 : Create View Page

Step 5 : Create Page for Update Records

Step 6 : Delete Records

Step 7 : Logout page

Also Read
SQL Join Types
By: Kyle Le
Tags: MySQL, Sql, Sql Server

A JOIN clause is used to combine rows from multiple tables, based on a related column between them.

There are four basic join types : inner, left, right and full

INNER JOIN : Returns records that have matching values in both tables

LEFT JOIN : Returns all records from the left table, and the matched records from the right table

RIGHT JOIN : Returns all records from the right table, and the matched records from the left table

FULL JOIN : Returns all records when there is a match in either left or right table

EXAMPLES OF SQL JOIN TYPES

Before going into the details, I want to introduce two tables that I’m going to use to demonstrate the examples. The relationship between them is specified by the ‘id’ primary key in the ‘customers’ table and the ‘customer_id’ foreign key in the ‘orders’ table.

the ‘customers’ table

the ‘orders’ table

INNER JOIN

For instance, if we want to get list of all customers who placed an order and details of the order they placed. Inner Join will be perfect for this demand since Inner Join returns all the records that match values in both tables. The syntax looks like this:

LEFT JOIN

If we just want the information of customers, regardless they placed an order or not, we would use LEFT JOIN which returns records from table left and any matching records from table right

Since Raiden, Hokage and Connie have not placed any order, the ‘amount’ and ‘date’ are NULL, simply means there is no data

RIGHT JOIN

RIGHT JOIN is sort of a opposite version of LEFT JOIN, in this case it would return all the orders with customer infomation attached

FULL JOIN

Finally, the FULL JOIN returns all records from both tables

Since FULL JOIN is not supported by MySQL, but we can emulate them using LEFT JOIN ,RIGHT JOIN and UNION. Try to figure out yourself.
Data-at-Rest encryption in MySQL
By: Shubham Khairnar
Tags: MySQL, Encryption, Data At Rest, Docker, Data

Setup:

The data-at-rest encryption feature relies on a keyring plugin for master encryption key management.

We will be using docker approach to achieve data at rest encryption in MySQL, so that you can save some time for snacks! 😜

Step 1:

Clone the repository:

*We are using MySQL version 8.0, yes we always use latest ones 😉

Step 2:

Setup a keyring plugin:

Move to conf/my.cnf

Add the following lines to my.cnf to setup keyring plugin:

early-plugin-load=keyring_file.so keyring_file_data=/var/lib/mysql/keyring-data/keyring innodb_file_per_table=ON

Step 3:

Build and run customized docker image to repository

Navigate to root folder

Build the docker image:

sudo docker build -t <IMAGE_NAME>:<VERSION_ID> .

Run the docker image:

sudo docker run -d — name mysql <IMAGE_NAME>:<VERSION_ID>

Step 4:

Execute the running docker container:

sudo docker exec -it mysql bash

Step 5:

After moving inside the container, check for /etc/mysql/my.cnf contains same configuration as we had set.

sudo vim /etc/mysql/my.cnf

Step 6:

Login to mysql instance:

mysql -uroot -p

*Default password is no password

Step 7:

Create database and tables:

mysql> CREATE DATABASE databasename; mysql> CREATE TABLE table_name ( column1 datatype, column2 datatype, column3 datatype, …. );

Step 8:

Insert some data:

mysql> INSERT INTO table_name (column1, column2, column3, …) VALUES (value1, value2, value3, …);

Step 9:

We have data in place, now we will move to encrypting this data at rest 😇

Check for the keyring plugin is in place:

If you didn’t get required output, please follow step:

mysql> INSTALL PLUGIN keyring_file SONAME ‘keyring_file.so’;

Step 10:

The default_table_encryption variable can be set for an individual client connection or globally using SET syntax.

The following statement enables default schema and tablespace encryption globally:

mysql> SET GLOBAL default_table_encryption=ON;

Step 11:

To encrypt particular table at-rest, first select the database:

mysql> use databasename;

Step 12:

To alter the encryption of an existing general tablespace, an ENCRYPTION clause must be specified:

mysql> ALTER TABLE table_name ENCRYPTION = ‘Y’;

Step 13:

Validate the encryption:

Our output should look like below:

Voilaa!!🥳 this is how we encrypted data at-rest in MySQL 😎
Building a connection between Java program and MySQL
By: Tharushi Chamalsha
Tags: Java, MySQL

When we are developing a java program which needs to store data, we should connect a database with the program. In this article we will learn the way of connecting MySQL database server along with java program.

I choose intellij IDEA development environment to run the java codes.

U should have,

Intellij IDEA development environment MySQL database management system

installed on your computer.

And also you should download the JDBC application programming interface in order to access the database. It can be use to build the java database connection. You may download the file using following link.

Then create a project using intellij idea and make a folder inside your project. Name it as lib and store the bin.jar file inside the lib folder which is extracted from downloaded JDBC file. Then go to the following path and export the bin.jar file to the dependencies.

File → Project Structure → Modules → Dependencies

And do as follows.

Now we can go see the way to build the connection between java program and the database.

First create a database and use its name to the URL.

Using the database and creating table.

Inserting records to the database

Updating new values to the database

Deleting record from database

Using above basic knowledge we can use other complex sql queries along with the java programs.
MySQL 8.0.22 | Create and Select View
By: Student Kim
Tags: MySQL

View — A virtual table based on the tables that already exist.

Hi guys! welcome to another MySQL session with me. Today, it’s View.

[View]

This is a virtual ‘Table’, so when you make it we use create statement, and when you delete it we use drop just like other tables. After making ‘view’ you can revise it, but it has a risk to cause error. So usually we use it only with select statement. The basic query of the ‘view’ is

create view view_name

as select select_list

from table_name where~

So basically, if you use certain select statement often on the tables, you can make the select into a virtual table ‘View’.

[Practice]

*****table1*****

create table member(

no int primary key auto_increment,

name varchar(5) not null,

addr varchar(2),

gender varchar(1) default 'w',

grade int default 1,

point int default 100); insert into member values(default,'Nina','NY','w',2,300);

insert into member values(default,'Tom','NY','m',1,200);

insert into member values(default,'Amy','LA','w',1,150);

insert into member values(default,'Jake','DC','m',2,100);

insert into member values(default,'Lisa','LA','w',2,300);

insert into member values(default,'Nicky','NY','w',3,400);

insert into member values(default,'Tim','LA','m',4,300); *****table2*****

create table mycat(

catno int primary key auto_increment,

userno int,

catName varchar(5),

catage int,

foreign key (userno) references member(no)); insert into mycat values (default,1,'kiki',2);

insert into mycat values (default,1,'lulu',2);

insert into mycat values (default,2,'tina',1);

insert into mycat values (default,3,'cici',1);

insert into mycat values (default,4,'toto',3);

insert into mycat values (default,4,'mio',1);

Ok, here I made these two tables. First one is the member, and the second one is their cats list.

Q. Create a view that shows members’ name and their cats’ name but only when the cat’s age is 1 year old.

Okay, this is a simple Join question. If you’re not familiar with the Join checkout my post below!

First we should select the right tuples.

mysql> select m.name, c.catName

-> from member m, mycat c

-> where m.no=c.userno and catage=1;

+------+---------+

| name | catName |

+------+---------+

| Tom | tina |

| Amy | cici |

| Jake | mio |

+------+---------+

3 rows in set (0.01 sec)

And then we’re making this select statement into a virtual table ‘view’.

mysql> create view catone

-> as select m.name, c.catName

-> from member m, mycat c

-> where m.no=c.userno and catage=1;

Query OK, 0 rows affected (0.01 sec) mysql> select * from catone;

+------+---------+

| name | catName |

+------+---------+

| Tom | tina |

| Amy | cici |

| Jake | mio |

+------+---------+

3 rows in set (0.01 sec)

Just like that, you can checkout what’s in the view with the same way to see the tuples in tables. And when you don’t need it anymore you can simply drop it.

mysql> drop view catone;

Query OK, 0 rows affected (0.01 sec)

That’s it for the view! From tomorrow, I’m gonna start to post the Java. So stay tuned! Thanks!
Membuat Restfull Api Menggunakan Express JS, ORM Sequelize dan MySQL Part 3: Login dan Register
By: Firman Abdul Hakim
Tags: MySQL, Restful Api, Sequelize, Expressjs, Nodejs

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Entity Developer and ADO.Net Data Providers Now Support Entity Framework Core 5
By: Jackson Peter
Tags: MySQL, Entity Framework Core, Postgresql, Sqlite, Oracle

Devart announced an update for Entity Developer and dotConnect ADO.NET data providers for Oracle, MySQL, PostgreSQL, and SQLite. The new versions now support Entity Framework Core 5.0.

Devart, a recognized vendor of professional database management software and world-class data connectivity solutions for various data connection technologies and frameworks, introduced support of the Entity Framework Core 5.0 in Entity Developer and dotConnect providers.

Entity Developer 6.10 can now detect many-to-many associations and table-per-type inheritances automatically when generating an Entity Framework Core 5.0 model from a database. Also, these inheritances and associations could be created manually via the corresponding editors or Model Refactoring Wizard.

Standalone Entity Developer application now targets Entity Framework Core 5.0 by default, when creating new Entity Framework Core models.

The newly released dotConnect ADO.NET data providers for Oracle, MySQL, PostgreSQL, and SQLite got significant changes to their NuGet packages with Entity Framework Core functionality.

These packages now contain new assemblies for the following Entity Framework Core versions:

EF Core 1 assembly (.NET Standard 1.3)

EF Core 3 assembly (.NET Standard 2.0)

EF Core 5 assembly (.NET Standard 2.1)

At the same time these packages do not contain an assembly for Entity Framework Core 2 anymore. This assembly is now available only via the provider installer.

For more information on how these changes may affect projects, using dotConnect providers and Entity Framework Core, please visit

https://blog.devart.com/entity-framework-core-5-support-in-entity-developer-and-dotconnect-providers.html

About Devart

Devart is one of the leading developers of database tools and administration software, ALM solutions, data providers for various database servers, data integration and backup solutions. The company also implements Web and Mobile development projects.

For additional information about Devart, visit https://www.devart.com/.
How To Create a Read-Only MySQL User
Tags: MySQL, Database, Readonly

Sometimes you need to create a read-only MySQL user credentials for a new team mate to prevent accidental queries being fired into your database. Here’s how you can create a read-only MySQL user.

1. Log into MySQL as an admin

Run the MySQL command-line program by doing one of the following:

a. At a UNIX prompt, run the MySQL command-line program, and log in as an administrator by typing the following command

mysql -u root -p

b. At a Windows command prompt, run the MySQL command-line program, and log in as an administrator by typing the following command from the MySQL root folder (e.g Program Files\MySQL\MySQL Server 5.5\bin):

mysql -u root -p

c. Run any database shell that works with MySQL. E.g, Heidisql and MySQL Workbench.

Type the password for root account.

2. Create a new MySQL user

Copy the following command and paste them into a MySQL shell. Replace $database_name, $user, and $password below with the values you will be using.

CREATE USER ‘$user‘@’127.0.0.1’ IDENTIFIED BY ‘$password‘;

3. Grant read-only permission to the MySQL user

Copy the following command and paste them into a MySQL shell.

GRANT SELECT, SHOW VIEW ON $database_name.* TO $user@’127.0.0.1′ IDENTIFIED BY ‘$password‘; FLUSH PRIVILEGES;

If you want to use SSL connection, you can use the following instead

GRANT SELECT, SHOW VIEW ON $database_name.* TO $user@’127.0.0.1′ IDENTIFIED BY ‘$password‘ REQUIRE SSL; FLUSH PRIVILEGES;

This creates a new read-only MySQL local user that can access your databases. If you want to create a remote user to access your database you can replace 127.0.0.1 with Remote IP. Also, you will need to enable remote access to your database. Please note, it is not a secure practice.

You can read more about how to create MySQL user here
DockerのMySQLで日本語が使えない！？
Tags: MySQL, Docker, 日本語

Dockerで起動したMySQLで日本語が入力出来ないんです。という質問があったので、内容の確認とその対応についてまとめてみました

$ docker run -e MYSQL_ROOT_PASSWORD='password' -d -p 3306:3306 mysql $ docker exec -it <CONTAINER ID> bash # mysql -u root -p

Enter password: password mysql> 日本語

とやると、「日本語」が消えてしまう（入力出来ない）というものでした

mysql> exit # 日本語

とやっても入力出来ません、これはMySQLコンテナが使っているイメージで、日本語入力が出来る設定がされていないからです

ログインしているコンテナで日本語が入力出来ないので、MySQLに接続しても日本語入力出来ません

コンテナに日本語入力に必要な設定をしてみます

Dockerfileを作成して以下の設定をします

#使うDockerイメージ

FROM mysql #日本語設定

RUN apt-get update && apt-get install -y locales

RUN sed -i -E 's/# (ja_JP.UTF-8)/\1/' /etc/locale.gen && locale-gen

ENV LANG ja_JP.UTF-8

設定内容としては、

apt-get update → パッケージ管理のapt-getに必要な情報を取得

apt-get install -y locales → 多国語サポート用ライブラリ（locales）インストール

sed -i -E ‘s/# (ja_JP.UTF-8)/\1/’ /etc/locale.gen→日本語入力に必要なja_JP.UTF-8がコメントアウト（無効化）されているので、コメントアウトを削除（有効化）する

locale-gen → ja_JP.UTF-8の設定を作成する（locale.genファイルで有効化したあとに、locale-genコマンド実行で設定が作成される）

ENV LANG ja_JP.UTF-8 → ja_JP.UTF-8を利用する宣言をする

設定が出来たらビルド、コンテナ起動して日本語入力を試してみます

$ docker build .

Sending build context to Docker daemon 2.048kB

Step 1/4 : FROM mysql

---> dd7265748b5d

Step 2/4 : RUN apt-get update && apt-get install -y locales

---> Using cache

---> e79295695ff9

Step 3/4 : RUN sed -i -E 's/# (ja_JP.UTF-8)/\1/' /etc/locale.gen && locale-gen

---> Using cache

---> b539a422d775

Step 4/4 : ENV LANG ja_JP.UTF-8

---> Using cache

---> 0a07da42c00e

Successfully built 0a07da42c00e $ docker run -e MYSQL_ROOT_PASSWORD='password' -d 0a07da42c00e $ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

d6567bd48620 0a07da42c00e "docker-entrypoint.s…" 29 seconds ago Up 29 seconds 3306/tcp, 33060/tcp busy_gates $ docker exec -it d6567bd48620 bash # echo 日本語

日本語

日本語入力が出来ました。続いてMySQLでも確認してみます

# mysql -u root -p

Enter password: password mysql> 日本語;

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '日本語' at line 1

同じく日本語入力の確認が出来ました
What is the between WHERE and HAVING clause in SQL and MySQL?
By: Sarthak Niwate
Tags: MySQL, Data Science, Sql, Data, Data Analysis

Adding a WHERE clause to a query allows you to set a condition which you can use to specify what part of the data you want to retrieve from the database.

HAVING is a clause frequently implemented with GROUP BY because it refines the output from records that do not satisfy a certain condition.

HAVING needs to be inserted between the GROUP BY and ORDER BY clauses. In a way, HAVING is like WHERE but applied to the GROUP BY block.

On some occasions, an identical result could be obtained by implementing the same condition, either with the WHERE or with the HAVING clause.

The main distinction between the two clauses is that HAVING can be applied for subsets of aggregated groups, while in the WHERE block, this is forbidden. In other words, after HAVING, you can have a condition with an aggregate function, while WHERE cannot use aggregate functions within its conditions.

Resourse: 365 Data Science
GCP Learning Series: connecting MySQL Client from Compute Engine
By: Prashanta Paudel, Prashantapaudel.Com.Np
Tags: MySQL, Compute Engine, Gcp Certification

Often you need to host a database in a cloud for different purposes. It is the most basic thing that needs to be done for any online software or apps. We already have computing capability in the cloud so, it will be a good idea to use compute instance to connect database instance for carrying out the task.

The process is consist of three steps

Setup Cloud SQL Setup Compute Instance Connect two instances

1 Setup Cloud SQL

First, goto GCP Console and click on SQL

Second, click on Create Instance

Select MySQL

Select the second generation, which is also mentioned as recommended.

Now input the instance name for your database, root password and the location of the database.

After a while, you will see that the database instance is ready. It generally takes few minutes so be patient. After the instance becomes ready you will notice green tick mark in its name.

2. Setup Compute Instance

Goto cloud console, click on Compute engine and then VM Instances

The idea here is to create a simple Linux machine with the capabilities of MySQL client.

Now that we have both machines ready, we can now install SQL by giving following commands

sudo apt-get update

sudo apt-get install mysql-client

Now, both our machines are ready. Let’s connect them.

3. Connect two instances

Firstly, go to compute instance and copy the public IP of the instance and paste that into SQL>connection>add network.

Now go to users and change the root password

After changing password head over to the compute instance and use the following command to connect to MySQL

# mysql -h (IP address of MySQL instance)-u root -p

Now you will see the screen as below which confirms that you have successfully setup database connection within the cloud
[MySQL]出現錯誤 1248 — Every derived table must have its own alias
Tags: MySQL, 1248, Error Handling, Database, Sql

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
REMOTE SYNC OF SUPERSET & MYSQL DOCKER CONTAINERS
Tags: MySQL, Data Science, Supersetapache, Data Visualization, Data

Prerequisite

Using Linux (Ubuntu 20.08)

As containers in docker:

Superset Apache

MySQL through MariaDB

You will learn

At the end of this article, you will learn how to remotely sync SupersetApache, a visualization tool, & MySQL database as docker containers remotely. This will enable access to data in MySQL and enable visual categorization and manipulation.

For easy sync of SupersetApache and the MySQL database as containers in docker, to fix the problem of change in IP address on your MySQL database with your desktop as a server:

Instead of:

mysql -u USERNAME -p

use:

mysql -h0.0.0.0 -uroot -p

Note that I am using MariaDB for my MySQL database.

With “root” as my username“-h0.0.0.0” reads the current comp server IP address.

Thus the SupersetApache MySql entry will thus be:

mysql+pymysql://root:XXXXXXXXXX@0.0.0.0:3306/mydb?charset=utf8
Level Up Your SQL Skills
By: Mandi Gunningham
Tags: Programming, MySQL, Database, Sql, Software Development

Level Up Your SQL Skills

Image from Caspar Camille Rubin on Unsplash.

According to JetBrains’ 2020 Developer Ecosystem Survey, 56% of developers utilize or interact with a SQL database in their development environment. Even so, most developers view SQL concepts as a secondary concern, as they are often interacting via their chosen language and do not play much of a part in the design. Generally speaking, being able to connect and run basic queries will get you through the day.

At many companies, there is even a separate database team that will be handling the database design and optimization, so you will never get first-hand exposure. I think this is often the deterrent to learning more about SQL. It’s viewed as a utility that is often handled by others. Software developers have enough on their plate, right? Fair enough.

However, in our industry, having full-stack skills is always a good idea. You never know what will be thrown at you. We interact with SQL daily, so more knowledge will always come in handy. Additionally, if you end up at a smaller company that does not have the resources for a full database team, more of those responsibilities will fall on you. Being able to state that you are confident in your database skills during an interview will open up many interesting career paths.

I personally enjoy SQL development more than most. I interned early in my career with a team of outstanding database administrators and quickly fell in love. I had taken my database courses in school, so I had some basic knowledge, but what I learned during that internship has been invaluable to me. Not only did it teach me that I have a secret passion for data, but the technical understanding I gained has been a boon in every job I’ve gotten going forward.

There is a lot to database development and administration — there are experts out there who can work magic. However, as a developer and not an SQL wizard, these are the topics that will set you apart from the pack when it comes to SQL knowledge.

Note: I have experience with Microsoft SQL Server, so that is what I will reference throughout. However, these concepts absolutely apply to other SQL DBMS options.
MySQL Window Function Compilation
By: Joshua Otwell
Tags: MySQL, Database, Technical Writing, Data, Data Analysis

Self-Promotion:

If you enjoy the content written here, by all means, share this blog and your favorite post(s) with others who may benefit from or like it as well. Since coffee is my favorite drink, you can even buy me one if you would like!

Without further ado, here is the compilation. Be sure and share the posts along with others’ who may value them as well:

Like what you have read? See anything incorrect? Please comment below and thank you for reading!!!

A Call To Action!

Thank you for taking the time to read this post. I truly hope you discovered something interesting and enlightening. Please share your findings here, with someone else you know who would get the same value out of it as well.

Visit the Portfolio-Projects page to see blog post/technical writing I have completed for clients.

Coffee is one of my favorite things to drink!

To receive email notifications (Never Spam) from this blog (“Digital Owl’s Prose”) for the latest blog posts as they are published, please subscribe (of your own volition) by clicking the ‘Click To Subscribe!’ button in the sidebar on the homepage! (Feel free at any time to review the Digital Owl’s Prose Privacy Policy Page for any questions you may have about: email updates, opt-in, opt-out, contact forms, etc…)

Be sure and visit the page for a collection of my best blog posts.

Josh Otwell has a passion to study and grow as a SQL Developer and blogger. Other favorite activities find him with his nose buried in a good book, article, or the Linux command line. Among those, he shares a love of tabletop RPG games, reading fantasy novels, and spending time with his wife and two daughters.

Disclaimer: The examples presented in this post are hypothetical ideas of how to achieve similar types of results. They are not the utmost best solution(s). The majority, if not all, of the examples provided, are performed on a personal development/learning workstation-environment and should not be considered production quality or ready. Your particular goals and needs may vary. Use those practices that best benefit your needs and goals. Opinions are my own.
IF Statements in MySQL, Conditional Logic
Tags: MySQL, Database, Db, Sql, If Statements

There are two different options when thinking of IF statements in MySQL

SELECT IF() function — Given a boolean expression, it returns a value depending on the result of the boolean expression IF statement — Given a boolean expression, it executes a certain block of code, depending on the result of the boolean expression. Can only be used inside a stored procedure.

SELECT IF() Function

Syntax

The syntax for the IF() function in MySQL is as follows:

SELECT IF (boolean_expression, return value if true, return value if false)

Example 1: Intro

Here’s an example:

SELECT IF (1 = 1, 'The expression is true', 'No it's false') # This returns: 'The expression is true'.

We pass the boolean expression 1 = 1 which of course is true, therefore our IF() function returns ‘The expression is true’.

Example 2: Realistic Example

Here’s a more realistic scenario as to where you might use an IF function:

SELECT

id,

IF (name == null, John Doe, name) name,

age,

gender

FROM

person;

In this example, we have a table person which holds id , name , age , and gender . If no name is present in our database, then we return John Doe.

Example 3: Aggregates

The IF() function can be used creatively. Consider this example, where we count the number of records which have a certain name:

SELECT

SUM( IF(person.name = 'Pradeega', 1, 0) ) AS pradeegas

FROM

person;

This would return a table which displays how many records have the name ‘Pradeega’:

|pradeegas |

+----------+

| 12|

+----------+

Example 4. Nested IFs

Since the IF() function takes SQL expressions as arguments, we can nest IF() functions.

SELECT

IF(person.name = 'Officer Johnson',

'Is a cop.',

IF(person.name = 'Ted Bundy', 'Is criminal', 'Unknown')

)

FROM

person;

Here we are essentially doing an “else-if” statement as found in many programming languages. If you have several values to check against, consider using a CASE function or a stored procedure + IF statement (not function).

IF Statements

IF statements are only available when used inside stored procedures and are for more complicated logic. See here for more details:

Reference

Thanks for read! Feel free to ask me any questions.
Instalaciones que necesitarás para empezar a usar WSL en tus proyectos
By: Ernesto Angulo
Tags: MySQL, Node, Wsl, Git, Linux

Just someone who likes design 🎨 and code 💻.
MySQL 8.0.22 | Select Subquery (Practice Questions 2)
By: Student Kim
Tags: MySQL

Hi guys! Welcome to another MySQL sesh with me. Continue from the last time let’s do some more practice on subquery! If you missed my last post, please click the link down below! You would really need it today…

[Practice Questions]

//Copy the Queries down below and solve the questions. create database shop;

use shop; create table member(

no int primary key auto_increment,

name varchar(5) not null,

addr varchar(2),

gender varchar(1),

grade int default 1,

point int default 100); insert into member values

(default,'Nina','NY','w',2,300),

(default,'Tom','NY','m',1,200),

(default,'Amy','LA','w',1,150),

(default,'Jake','DC','m',2,100),

(default,'Lisa','LA','w',2,300),

(default,'Nicky','NY','w',3,400),

(default,'Tim','LA','m',4,300);

Q1. Return the average point of each grade, but only when it’s higher than the total average.

Just like the last time, I’ll divide this question into two select statement.

1.

select grade, avg(point) from member group by grade having (?????) 2.

select avg(point) from member

Do you see which one should be the subquery here? That’s right. it’s the second one!

Answer : mysql> select grade, avg(point)

-> from member

-> group by grade

-> having avg(point)>(select avg(point) from member);

+-------+------------+

| grade | avg(point) |

+-------+------------+

| 3 | 400.0000 |

| 4 | 300.0000 |

+-------+------------+

2 rows in set (0.00 sec)

This time I put the subquery in the having clause to give a certain condition on the group.

Q2. Select the people who has the lower point than ‘Tom’.

This one is pretty easier than other subquery practices. In this case we should get the point of ‘Tom’ first. And make it into subquery.

1. Subquery

select point from member where name ='Tom' 2. Main query

select name, point from member where point ???

A-ha, Now we need to put the subquery into the main query!

Answer : mysql> select name, point

-> from member

-> where point<(select point from member where name='Tom');

+------+-------+

| name | point |

+------+-------+

| Amy | 150 |

| Jake | 100 |

+------+-------+

2 rows in set (0.00 sec)

Just like this.

Well done today guys! In the next session I’ll be talking about the view. Thanks!
Mysql Database Replication
By: Myoo Thu
Tags: MySQL, Replication, Devnote

mysql replication ဆိုတာ ကိုယ့်ရဲ ဒေတာဘေစ်ကို copy ပြုလုပ်ချင်းပါပဲ။

အကြောင်းအမျိုးမျိုးကြောင့် ကိုယ့်ရဲ့ ဒေတာဘေစ် ပျက်စီးခဲ့မယ်ဆိုရင် copy လုပ်ထားတဲ့ ဒေတာဘေစ်တွေ

ရှိတဲ့အတွက် ဒေတာဆုံးရှုံးမူတွေလည်း သက်သာနိုင်ပါတယ်။ Performance ဘက်ကကြည့်မယ်ဆိုရင်လည်း

Read Query တွေကို replicate လုပ်ထားတဲ့ ဒေတာဘေစ်တွေကို ရွှေ့နိုင်ပြီး Master ဒေတာဘေစ်မှာ

loading ကိုသက်သာစေနိုင်ပါတယ်။ Replication လုပ်ခြင်းက read performance ကိုပဲ သက်သာစေနိုင်ပြီး

Write performance ကိုတော့ လျှော့ချစေနိုင်မှာမဟုတ်ပါဘူး။

Master Database crashed ဖြစ်သွားခဲ့တယ်ဆိုရင်လည်း slave database ကို master အဖြစ်အသုံးပြုလို့ရတဲ့အတွက်

ကိုယ့်ရဲ့ system ကို အမြန်ဆုံး ပုံမှန်အခြေအနေရောက်အောင် ပြုလုပ်ပေးနိုင်ပါတယ်။

Mysql Replication မှာ type နှစ်မျိုးရှိပါတယ်

Bin Log Replication

GTID based replication

GTID replication ကို mysql 5.6 ကျမှsupport ပေးထားတာဖြစ်ပါတယ်။ GTID ရဲ့ပုံစံက

transaction တိုင်းမှာ global unique id ရှိပြီးတော့ slave server တွေက အဲ့ဒီ ID နဲ့ပဲ slave server မှာ ဒေတာတွေ update လုပ်တာဖြစ်ပါတယ်။

Bin Log replication ကတော့ master မှာရှိတဲ့ binary log တွေကို မူတည်ပြီး salve ဆာဗာတွေမှာ ဒေတာတွေကို replicate လုပ်တာဖြစ်ပါတယ်။

ဒါကြောင့် Slave ဆာဗာတွေက master ရဲ့ bin log position ကိုအမြဲသိရဖို့ လိုတာဖြစ်ပါတယ်။

နောက်ပြီး

Mysql Replication မှာ Mode သုံးမျိုးရှိပါရှိပါသေးတယ်။

Statement Base Replication

Row Base

Mix



State Base Replication ကတော့ sql statement အလိုက် replication လုပ်တာဖြစ်ပါတယ်။

row base replication ကတော့ row တစ် row ဆီအပေါ်မှာ အခြေခံပြီးတော့

replication လုပ်တာဖြစ်ပါတယ်။ အကုန်လုံးမှာ အားသာချက်အားနည်းချက်တွေရှိပါတယ်။

Bin Log Replication ပြုလုပ်ပုံလေးကို ပြောပြပေးပါမယ်။

ပထမဆုံး Replication လုပ်ဖို့ server နှစ်ခု မှာ ဒေတာဘေစ် နှစ်ခုလိုပါတယ်။ တစ်ခုက master database ဖြစ်ပြီး

နောက်တစ်ခုကတော့ slave database ပဲဖြစ်ပါတယ်။ ကျွန်တော်တို့တကယ်အသုံးပြုနေတဲ့ ဒေတာဘေစ်က

master database ပဲဖြစ်ပါတယ်။ Slave Database ကတော့ ကျွန်တော်တို့ Copy လုပ်ထားတဲ့ ဒေတာဘေစ်ပဲဖြစ်ပါတယ်။

Master Database မှာ slave database တစ်ခုထက်မက ထားလို့ရပါတယ်။ slave database ကိုတော့ master database

အများကြီး သုံးလို့မရပါဘူး။

အောက်က ip address တွေနဲ့ Server နှစ်ခုရှိတယ်ဆိုပါဆို့

12.34.56.789- Master Database

98.76.65.432- Slave Database

ပထမဆုံး server နှစ်ခုစလုံးမှာ mysql database server သွင်းရပါမယ်

sudo apt-get install mysql-server mysql-client

ပြီးရင် master server က mysql config မှာ ပြင်ပေးရပါမယ်

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

bind-address = 127.0.0.1

bind address နေရာမှာ ကျွန်တော် local address ဖြစ်နေပါတယ် အဲ့နေရာမှာ ip address ဖြစ်ရပါမယ်။

bind-address = 12.34.56.789

service -id နဲ့ log_bin ကို uncomment လုပ်ပေးပါ။

log_bin = /var/log/mysql/mysql-gin.log

server-id=1

ပြီးရင် ကျွန်တော်တို့ replication လုပ်မဲ့ ဒေတာဘေစ်ကို သတ်မှတ်ပေးရပါမယ်။

binlog_do_db = testdb

ပြီးရင် mysql restart ချဖို့လိုပါမယ်

sudo service mysql restart

mysql database ကို ဝင်ပါ

mysql -u root -p

user ဆောက်ပြီး permission ပေးပါမယ်။

CREATE USER ‘slaveUser’@’98.76.65.432' IDENTIFIED BY ‘YOUR_PASSWORD’;GRANT REPLICATION SLAVE ON *.* TO ‘slaveUser ‘@’98.76.65.432';
Mysql varchar max length
By: Suresh Kamrushi
Tags: MySQL, Mysql 8, Mysql Server

Maximum data what we can store in varchar in mysql was 255. But after Mysql version 5.0.3 varchar can store maximum of 65,535 characters. But again this limit is having limitation of maximum row size which is 65535 bytes. It means including all columns it should be less than 65,535 bytes. Lets take an example as below: I have created below table with two columns as “f1” varchar with the length of 32,765 and “f2” with 32766. Total length will be 32765+2(for storing length) + 32766 + 2 = 65535.

CREATE TABLE IF NOT EXISTS `test_table` (

`f1` varchar(32765) NOT NULL,

`f2` varchar(32766) NOT NULL

) ENGINE=MyISAM DEFAULT CHARSET=latin1;

But when I create increase the column length with a single character as below:

CREATE TABLE IF NOT EXISTS `test_table2` (

`f1` varchar(32766) NOT NULL, //INCREASED LENGTH BY 1

`f2` varchar(32766) NOT NULL

) ENGINE=MyISAM DEFAULT CHARSET=latin1;

it will give me error.

#1118 - Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs

For more details as below: dev.mysql.com

Hope it help somebody out there….
MySQL Group Concat Function
By: Praj Basnet
Tags: MySQL, Database, Mariadb

The group_concat() function in MySQL is really nifty. It concatenates all rows that match your group by statement into a single, comma separated string.

Essentially, it transposes rows to columns.

Take this example. Say you have a table that stores a set of locations by country, state and city.

You could then write SQL as follows:

select

country,

state,

group_concat(city) as cities

from locations

group by country, state

What this is saying is bring back 3 columns, country, state, and for each combination of country and state, a list of all of the cities in that country and state as the third column (called cities).

This is actually a phenomenally handy thing to be able to do and something you’ll use a lot once you know it exists!
At Capillary Technologies, we used scripts to alter big tables of 500GB and bigger .
Tags: MySQL, Aurora, Alter Table

At Capillary Technologies, we used scripts to alter big tables of 500GB and bigger . We expect all tables to have autoincrement columns and a timestamp with ON UPDATE CURRENT_TIMESTAMP.

Reference: https://github.com/georgecj11/hotswap_mysql_table

Also using Aurora cluster which uses storage level sharing instead of master-slave are better to avoid lag and to guarantee consistency.

The solution you are using might not work if row based replication is used.
What is the Stored Procedure in MySQL? learn easy in 2 steps.
By: Sumant Jha
Tags: MySQL, Stored Procedure, Sql, Mysql Server, Sql Server

A procedure is also called a stored procedure in MySQL. It is a subroutine (like a subprogram) in a regular scripting language, which is stored in a database. In the case of MySQL, the procedures are written in MySQL and stored in the MySQL database/server. A MySQL procedure consists of a name, a parameter list, and a SQL statement (s).

The reason to use stored procedure in MySQL?

Stored procedures in MySQL are faster. MySQL takes some advantage of server caching, as do ready statements. The main speed advantage comes from the decrease in network traffic. If you have a repetitive task with no checking, looping, multiple statements, and no user interaction, do this with a call to a process that is stored on the server.

Stored procedures are portable. When you write your stored procedure in SQL, you know that it will run on every platform on which MySQL runs, forcing you to install additional run-environment packages or program execution in the operating system. Sets permissions for, or deploys different packages. If you have different computer types. This has the advantage of writing in SQL rather than in an external language like Java or C or PHP.

Stored procedures are basically available as ‘source code’ in the database itself. And it makes sense to link data to processes that run on data.

Stored Procedures Are Migrant! MySQL follows quite closely the SQL. Others (DB2, Oracle) also follow.

There are different types of MySQL Stored Procedure:
Top n Window Function queries in MySQL
By: Joshua Otwell
Tags: MySQL, Database, Technical Writing, Data, Data Analysis

Self-Promotion:

If you enjoy the content written here, by all means, share this blog and your favorite post(s) with others who may benefit from or like it as well. Since coffee is my favorite drink, you can even buy me one if you would like!

Here is the description of the target table I am working with for this blog post. If you have read any of my other posts, you may recognize this table as I tend to use it widely for example queries. The ‘walking_stats’ table stores my personal walking data, that I track while working on my fitness and to maintain a healthier weight:

There are just over 400 rows of data in the table ‘walking_stats’:

Top n Window Function query: Top 3 days

I store the amount of time — in hours, minutes, and seconds — spent walking in the ‘duration’ column. Suppose I want to find the ‘top 3’ number of rows — meaning the longest walks time-wise — over a 6-month span, broken down by each month? This type of question can be easily answered using Top n Window Function queries.

I’ll make use of a CTE instead of a subquery or derived table and generate the rows based on a WHERE clause that restricts the results to the year 2020 within the months of January to June.

Let’s look at the overall query:

Below are the query results:

In the screen-shot above, I have segregated the groups of rows using red rectangles to create some distinction between each set of top 3 duration rows for each month.

So how does this query work?

Top n Window Function query: Ranking and partitioning data
MySQL / MariaDB Storage Engine
By: Praj Basnet
Tags: MySQL, Database, Mariadb, Sysadmin

To see what storage engine is used by the server by default you can use the following command in the MySQL/MariaDB shell:

show variables like '%storage_engine';

You can view a list of supported engines using:

show engines\G;

To find out what storage engine your tables are using by schema use this command. You might want to filter by TABLE_SCHEMA to see a specific database.

select TABLE_SCHEMA, TABLE_NAME, ENGINE

from information_schema.tables;

You can view statistics about a particular engine using the following (adjust for the relevant engine):