Bug: Cassandra-Stress : Performance degraded with Cassandra on Single node cluster
Status: Resolved
Priority: Normal
Resolution: Invalid
Created: 20/Apr/20 13:41
Resolved: 20/Apr/20 16:22

Steps to recreate:
 # I have created RAID 0 with 8 NVMe disks and created a ext4 filesystem (800GB volume)
 # Replace all default storage /var/lib/cassandra to " /mnt" in cassandra.yaml file.
 # Run cassandra-stress tool for performance benchmarking, am getting lower op/s than Aerospike NoSQL. 

SSD Drive is capable of 

Read Throughput (max MiB/s, 128KiB) : 3.1K

Read Throughput (max MiB/s, 128KiB) : 1.8K

Read IOPS (max, Rnd 4KiB) : 467K

Read Throughput (max, Rnd 4KiB) :65K
{noformat}
root@cassandra-master:~# cassandra-stress write n=1000000 -rate threads=64
******************** Stress Settings ********************
Command:
  Type: write
  Count: 1,000,000
  No Warmup: false
  Consistency Level: LOCAL_ONE
  Target Uncertainty: not applicable
  Key Size (bytes): 10
  Counter Increment Distibution: add=fixed(1)
Rate:
  Auto: false
  Thread Count: 64
  OpsPer Sec: 0
Population:
  Sequence: 1..1000000
  Order: ARBITRARY
  Wrap: true
Insert:
  Revisits: Uniform:  min=1,max=1000000
  Visits: Fixed:  key=1
  Row Population Ratio: Ratio: divisor=1.000000;delegate=Fixed:  key=1
  Batch Type: not batching
Columns:
  Max Columns Per Key: 5
  Column Names: [C0, C1, C2, C3, C4]
  Comparator: AsciiType
  Timestamp: null
  Variable Column Count: false
  Slice: false
  Size Distribution: Fixed:  key=34
  Count Distribution: Fixed:  key=5
Errors:
  Ignore: false
  Tries: 10
Log:
  No Summary: false
  No Settings: false
  File: null
  Interval Millis: 1000
  Level: NORMAL
Mode:
  API: JAVA_DRIVER_NATIVE
  Connection Style: CQL_PREPARED
  CQL Version: CQL3
  Protocol Version: V4
  Username: null
  Password: null
  Auth Provide Class: null
  Max Pending Per Connection: 128
  Connections Per Host: 8
  Compression: NONE
Node:
  Nodes: [localhost]
  Is White List: false
  Datacenter: null
Schema:
  Keyspace: keyspace1
  Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
  Replication Strategy Pptions: {replication_factor=1}
  Table Compression: null
  Table Compaction Strategy: null
  Table Compaction Strategy Options: {}
Transport:
  factory=org.apache.cassandra.thrift.TFramedTransportFactory; truststore=null; truststore-password=null; keystore=null; keystore-password=null; ssl-protocol=TLS; ssl-alg=SunX509; store-type=JKS; ssl-ciphers=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA; 
Port:
  Native Port: 9042
  Thrift Port: 9160
  JMX Port: 7199
Send To Daemon:
  *not set*
Graph:
  File: null
  Revision: unknown
  Title: null
  Operation: WRITE
TokenRange:
  Wrap: false
  Split Factor: 1


WARN  19:01:48,641 You listed localhost/0:0:0:0:0:0:0:1:9042 in your contact points, but it wasn't found in the control host's system.peers at startup
Connected to cluster: Test Cluster, max pending requests per connection 128, max connections per host 8
Datatacenter: datacenter1; Host: localhost/127.0.0.1; Rack: rack1
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Warming up WRITE with 50000 iterations...
Running WRITE with 64 threads for 1000000 iteration
type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
total,        120763,  120763,  120763,  120763,     0.5,     0.4,     1.0,     2.1,    24.9,    29.1,    1.0,  0.00000,      0,      1,      60,      60,       0,    1499
total,        254564,  133801,  133801,  133801,     0.5,     0.3,     1.0,     1.8,     8.7,    64.5,    2.0,  0.04931,      0,      0,       0,       0,       0,       0
total,        363914,  109350,  109350,  109350,     0.6,     0.3,     0.9,     1.7,    20.9,   254.7,    3.0,  0.04089,      0,      1,     251,     251,       0,    1470
total,        514894,  150980,  150980,  150980,     0.4,     0.3,     0.8,     1.6,     4.9,    21.8,    4.0,  0.03878,      0,      1,     107,     107,       0,    1466
total,        665446,  150552,  150552,  150552,     0.4,     0.3,     0.8,     1.6,     4.8,   109.4,    5.0,  0.04428,      0,      0,       0,       0,       0,       0
total,        788349,  122903,  122903,  122903,     0.5,     0.3,     1.0,     1.7,    27.5,   110.4,    6.0,  0.03855,      0,      1,     106,     106,       0,    1480
total,        915193,  126844,  126844,  126844,     0.5,     0.3,     1.0,     1.7,     3.8,   109.4,    7.0,  0.03331,      0,      1,     107,     107,       0,    1477
total,       1000000,  128730,  128730,  128730,     0.5,     0.4,     1.0,     1.6,     4.3,    20.6,    7.7,  0.03118,      0,      0,       0,       0,       0,       0


Results:
Op rate                   :  130,569 op/s  [WRITE: 130,569 op/s]
Partition rate            :  130,569 pk/s  [WRITE: 130,569 pk/s]
Row rate                  :  130,569 row/s [WRITE: 130,569 row/s]
Latency mean              :    0.5 ms [WRITE: 0.5 ms]
Latency median            :    0.3 ms [WRITE: 0.3 ms]
Latency 95th percentile   :    0.9 ms [WRITE: 0.9 ms]
Latency 99th percentile   :    1.7 ms [WRITE: 1.7 ms]
Latency 99.9th percentile :    6.3 ms [WRITE: 6.3 ms]
Latency max               :  254.7 ms [WRITE: 254.7 ms]
Total partitions          :  1,000,000 [WRITE: 1,000,000]
Total errors              :          0 [WRITE: 0]
Total GC count            : 5
Total GC memory           : 7.219 GiB
Total GC time             :    0.6 seconds
Avg GC time               :  126.2 ms
StdDev GC time            :   65.0 ms
Total operation time      : 00:00:07 {noformat}
 

Please let me know if you need more information