Redshfit の classic resize の仕組み

業務で調べることがあったのでメモる。あまりまとまってないのでいつか編集します。

# Enhanced Classic Resizing

[2022/07](https://aws.amazon.com/blogs/big-data/accelerate-resizing-of-amazon-redshift-clusters-with-enhancements-to-classic-resize/)に発表されたEnhanced Classic Resizing の仕組み。

1. Create a new cluster as the destination cluster.
2. Make source cluster read-only.
3. Copy all data from source to destination without restoring to original distribution key and sort key. All data is set to diststyle Even on destination cluster.
4. Delete the source cluster and Flip over the destination cluster as current cluster. 
5. Cluster is available to read and write. Background process starts to redistribute/sort data. It would take very long time to restore to original distribution/sort keys. During this process, Since lots of  tables are distributed Even,  Queries would be performed much slower than normal. It would expect to be significant long time that queries run at down-graded performance. After all the table restored to original distribution and sorting key, queries would perform normally.

# 旧式 Classic Resizing

なお、場合によっては、今でも旧式メカニズムが使われる。例えば、
- 「Node数とNodeタイプに変更を加えずにresizingを起動した場合
- ターゲットタイプがDC2/DS2の場合

https://aws.amazon.com/blogs/big-data/accelerate-resizing-of-amazon-redshift-clusters-with-enhancements-to-classic-resize/


1. Create a new cluster as the destination cluster.
2. Make source cluster read-only.
3. Transfer data from source cluster to destination cluster. Please note that this process is expected to take long time, data from source cluster would lost its data distribution/sorting and in destination cluster it would restore to data original distribution/sorting setting. Data transfer though network would not take much time, But restoring original distribution/sorting is a time-consuming process in destination cluster. Even the same node number/type classic resize would follow this data re-distribution process as it is a classic resize.

4. When Restoring original distribution/sorting finished on destination cluster, the source cluster is deleted and the destination cluster is  flipped over as current cluster. 

5. Now, the classic resize process finished, and the cluster is available to read and write.

# おまけ: Classic Resize vs Elastic Resize

> For best practice, classic resize (normal/enhanced) is suitable for workload permanent change. For example, your ETL job numbers increasing/decreasing or you need more/less storage. If you need bring temporary power to deal with peak traffic, for example, nightly ETL, I would recommend elastic resize[2], which is quick to deploy (usually under 20 mins). 

# Ref.

<details>
<summary>AWSサポートチケットでの会話</summary>

Same node number/type classic resize is actually allowed and it would follow the same classic resize process:

1. Create a new cluster as the destination cluster.

2. Make source cluster read-only.

3. Transfer data from source cluster to destination cluster. Please note that this process is expected to take long time, data from source cluster would lost its data distribution/sorting and in destination cluster it would restore to data original distribution/sorting setting. Data transfer though network would not take much time, But restoring original distribution/sorting is a time-consuming process in destination cluster. Even the same node number/type classic resize would follow this data re-distribution process as it is a classic resize.

4. When Restoring original distribution/sorting finished on destination cluster, the source cluster is deleted and the destination cluster is  flipped over as current cluster. 

5. Now , the classic resize process finished, and the cluster is available to read and write.

This guide[1] specified an enhancement approach to classic resize:

1. Create a new cluster as the destination cluster.

2. Make source cluster read-only.

3. Copy all data from source to destination without restoring to original distribution key and sort key. All data is set to diststyle Even on destination cluster.

4. Delete the source cluster and Flip over the destination cluster as current cluster. 

5. Cluster is available to read and write. Background process starts to redistribute/sort data. It would take very long time to restore to original distribution/sort keys. During this process, Since lots of  tables are distributed Even,  Queries would be performed much slower than normal. It would expect to be significant long time that queries run at down-graded performance. After all the table restored to original distribution and sorting key, queries would perform normally.

Under normal/enhanced classic resize, Data redistribution time is  largely determined by data size and destination node type and numbers. According to document[1], when a 10TB cluster classic resize to 2 nodes of ra3.4xlarge, it took 738 minutes (12 hours). So for your 90TB cluster classic resize to 2 nodes of ra3.4xlarge , it would take 12*9=108 hours (5.5 days). If you cluster resize to 4 nodes of ra3.4xlarge, it may take less time (3-4 days) as this destination has more slices to run the data distribution. Source cluster data distribution is another factor, the more tables originally set to EVEN/ALL diststyle, the quicker data redistribution process would be on the assumption of same data size and same destination node type/number.

For best practice, classic resize (normal/enhanced) is suitable for workload permanent change. For example, your ETL job numbers increasing/decreasing or you need more/less storage. If you need bring temporary power to deal with peak traffic, for example, nightly ETL, I would recommend elastic resize[2], which is quick to deploy (usually under 20 mins). 


As agreed, I marked this case as resolved. Please feel free to contact AWS Support if you have any questions in the future.


Reference:
[1] https://aws.amazon.com/blogs/big-data/accelerate-resizing-of-amazon-redshift-clusters-with-enhancements-to-classic-resize/ 
[2] https://docs.aws.amazon.com/redshift/latest/mgmt/managing-cluster-operations.html#elastic-resize 

</details>


https://aws.amazon.com/blogs/big-data/accelerate-resizing-of-amazon-redshift-clusters-with-enhancements-to-classic-resize/