Package: cassandra-toolbox
cassandra-toolboxA suite of tools for Cassandra - A highly scalable distributed NoSQL datastore.These are a set of monitoring and analysis tools that were initially developed at Knewton in order to introspect behavior and performance of the Cassandra clusters at knewton.  These scripts are designed to run locally or remotely against a cassandra cluster.  Just install this into your virtual environment and go for it.Be aware that these scripts may cause some load on the target server which can impact your performance.  Nothing comes without a price.InstallationIt is preferable to create a virtual environment to install these tools intopipinstallvirtualenvvirtualenv/path/to/your/virtualenvsource/path/to/your/virtualenv/bin/activateTo install the project you can clone this repo and install it directly into your virtual environment, this would get you the latest codegitclonegit@github.com:Knewton/cassandra-toolbox.gitcdcassandra-toolbox./setup.pyinstallYou can also directly install our package from pippipinstallcassandra-toolboxUsageThis package consists of several scripts that get installed as executables into the environment that the package is in.  Each script does something different and is described independently below.cassandra-statThe cassandra-stat tool shows a real-time feed of how many operations have gone through Cassandra as reported by the JMX interface that is exposed.  Many people pump this information through Graphite or another data collector which is a good idea so you can visualize and store the data.  However these systems usually poll every minute and can lag behind events.  This allows you to see to the second what is occuring on the system in a manner similar to iostat.This script interfaces with the jolokia jmx-http bridge which must be attached to your Cassandra instances that you plan to run cassandra-stat against.  This is very easy however and not intrustive to Cassandra.Installing Jolokia is painless.  You firstLink download the JVM agent from their website <https://jolokia.org/download.html>.  Then you modify yourcassandra-env.shfile to include the following line:JVM_OPTS=”$JVM_OPTS -javaagent:<Path To Jolokia JVM Agent Jar>.jar”A restart of the Cassandra instance is required after this modification.To use cassandra-stat you only need to execute the following local to the Cassandra instance you wish to monitor.  You must have activated the virtual environment that cassandra-toolbox is installed into if you have installed the pacakge into a virtual environment.  The output will look similar to the following:$cassandra-statReadsRangesWritesReads(99%)msRanges(99%)msWrites(99%)msCompactionsFlushesRowCacheMissesTimens1011191.46268.8117.400020:15:36total2411391.468.3017.9800020:15:37total0011791.468.3017.1700020:15:38total007291.468.3017.3400020:15:39total026991.468.3017.300020:15:40totalThe fields that are output are as follows:Reads:               Number of reads since the last line was reportedRanges:              Number of range queries since the last line was reportedWrites:              Number of writes (updates/insertions/deletions) since the last line was reportedReads (99%):         99th percentile latency in reads given in millisecondsRanges (99%):        99th percentile latency in range queries given in millisecondsWrites (99%):        99th percentile latency in writes (updates/insertions/deletions) given in millisecondsCompactions:         Number of pending compactionsFlushes :            Number of memtable flushes that are in progressRow Cache Misses:    Number of row cache misses that have occured since the last line was reportedtime:                Time in HH:MM:SS that the line was recorded atns:                  Namespace that the statistic is output for.  This can be “total” which is a sum for all keyspaces,<keyspace>which is a sum of all column families inside that keyspace, or<keyspace>.<columnfamily>which is the most granular outputIf a namespace has no traffic, that is if there are 0 reads, range queries, or writes reported for that namespace then the namespace will not be output to the screen with the exception of “total” which will always be output.  Note that some of the statistics are differences with the previous line so the absolute numbers can vary depending on the rate that is chosen (this is configurable, see below).  Additionally the 99 percentile outputs are for a moving average that is generated internally by the Cassandra metric libraries so they are not representative of the 99th percentile at that instant.  Any aggregate level metrics (total or keyspace level metrics) will show the highest 99th percentile latency.  Aggregates for other metrics are summations of the constituent column families.  By default system keyspaces are not included in these aggregations, but this is configurable.Configurations are set by command line flags, these can be accessed by runningcassandra-stat--help:--header_rowsint:An integer of how many rows should pass before a new header line is output.  If this is 0 then only the first header will be printed, and if this is -1 then the top header row will not be printed.  Default 10.--rateint:How many seconds should pass between server polls.  Default 1.--show_system:Include system keyspaces and their related column families in the output.  The aggregation in “total” will include system keyspace entires as well, which is not the default behavior.--hoststring:Thehttp://HOST:PORTthat this script should connect to.  Defaulthttp://localhost:8778.--show_keyspaces:Set this flag in order to show keyspace level output.--show_cfs:Set this flag to show<keyspace>.<columnfamily>level output.--no_total:Set this flag to suppress output of the aggregated total.  This may have the effect of having no output when there is no traffic in the database.--show_zeros:Show all namespaces that are set to be output regardless if there is no activity.--namespacesstring:Comma separated list of namespaces to process and show.  To show an entire keyspace then use the keyspace name as an entry, and to show only a column family then add<keyspace>.<columnfamily>to the list.  For example--namespaceskeyspace1,keyspace2.test1would show all column families fromkeyspace1and only thetest1column family fromkeyspace2.  Note that there can be strange behaviors with this flag as other flags.  Adding a system keyspace to this flag will have no effect unless the--show_systemflag is also included.  Adding namespaces to this flag will change what namespaces are included in the total for aggregations but these namespaces will not be printed out if the--show_keyspacesor--show_cfsflags are not included.  For example--namespaceskeyspace1,keyspace2.test1--show_keyspaceswould output an entry fortotal,keyspace1, andkeyspace2wheretotalwill be the aggregate of all column families inkeyspace1as well as the column familytest1inkeyspace2,keyspace1will be the aggregate of all column families within it, andkeyspace2will be the aggregate of only the column familytest1.  If you are using this flag and--show_systemthen you must include which system keyspaces you wish to include, they will not be included by default if using the--namespacesflag.cassandra-tracingThis script facilitates analysis of Cassandra tracing data, which on its own is difficult to draw conclusions from.  Tracing is very useful to look at individual CQL queries in the cqlsh shell, however to get a better idea of your system’s behavior it is often useful to sample a percentage of queries, trace them, and save these traces for later.  Cassandra makes this easy to do with nodetool.  You can use the following to set the probability that any cql query (that is coordinated by the cassandra node local to this command) is traced to 0.1%.$nodetoolsettraceprobability0.001Note that a value of 1 means every query will be traced.  A trace logs many lines for every query so be careful of setting this value very high.  It is possible that tracing less than 5% of all queries can result in a write workload which is equal in number to the load which it is tracing.To use cassandra-tracing you only need to execute the following local to the Cassandra instance you wish to investigate the tracing logs of.  You must have activated the virtual environment that cassandra-toolbox is installed into if you have installed the pacakge into a virtual environment.  The output will look similar to the following:$cassandra-tracing`hostname-I`100%Complete:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX|100Totalskippedduetonullduration:0Totalskippedduetoerror:0175sessionssatisfyingcriteria.Showing100longestrunningresults.SessionIdDuration(microsecs)MaxTombstonesTotalTombstonesFlagsTimeStartedQuerybedd77a0-159e-11e6-af1d-5b2aec1d09441969632322016-05-09T04:30:23.259000SELECT*FROMsystem.schema_columnfamiliesbe941af0-1801-11e6-9deb-25e4276011e420569002016-05-12T05:24:05.280000Executingsingle-partitionqueryonColumnFamilyAb4c924a0-1565-11e6-af1d-5b2aec1d09442090532322016-05-08T21:42:05.041000SELECT*FROMsystem.schema_columnfamiliesfdf6fda0-174a-11e6-ace9-39442dcd207a21056002016-05-11T07:35:53.724000Executingsingle-partitionqueryonColumnFamilyBc5f24e80-1751-11e6-af1d-5b2aec1d094421397002016-05-11T08:24:26.216000Executingsingle-partitionqueryonColumnFamilyBf7670600-183b-11e6-af1d-5b2aec1d094421992002016-05-12T12:20:51.425000Executingsingle-partitionqueryonColumnFamilyCThe output displays the following information, sorted by duration:Session IDDuration in microsecondsMax number of tombstones encountered on a single sessionTotal tombstones encounteredFlags signifying special behavior, according to the legend:R = read repairT = timeoutI = index usedStarting timestamp of the session (hidden in slim mode)CQL query or inferred query fragment (hidden in slim mode)The Session Id can be used to introspect specific queries (which are logged as a session) in cqlsh yourself, by querying for the session id in the events table of the system_tracing keyspace.$cqlsh`hostname-I`Connectedtocassandraat172.ip.ip.ip:9042.[cqlsh5.0.1|Cassandra2.1.11|CQLspec3.2.1|Nativeprotocolv3]UseHELPforhelp.cqlsh>usesystem_traces;cqlsh:system_traces>select*fromeventsWHEREsession_id=bedd77a0-159e-11e6-af1d-5b2aec1d0944;session_id|event_id|activity|source|source_elapsed|thread--------------------------------------+--------------------------------------+--------------------------------------------------------------------------------------------------+--------------+----------------+---------------------bedd77a0-159e-11e6-af1d-5b2aec1d0944|bedd9eb0-159e-11e6-af1d-5b2aec1d0944|ParsingSELECT*FROMsystem.schema_columnfamilies|172.ip.ip.ip|21|SharedPool-Worker-2bedd77a0-159e-11e6-af1d-5b2aec1d0944|beddecd0-159e-11e6-af1d-5b2aec1d0944|Preparingstatement|172.ip.ip.ip|31|SharedPool-Worker-2bedd77a0-159e-11e6-af1d-5b2aec1d0944|bede13e0-159e-11e6-af1d-5b2aec1d0944|Computingrangestoquery|172.ip.ip.ip|73|SharedPool-Worker-2bedd77a0-159e-11e6-af1d-5b2aec1d0944|bede3af0-159e-11e6-af1d-5b2aec1d0944|Submittingrangerequestson1rangeswithaconcurrencyof1(22.37143rowsperrangeexpected)|172.ip.ip.ip|88|SharedPool-Worker-2bedd77a0-159e-11e6-af1d-5b2aec1d0944|bede6200-159e-11e6-af1d-5b2aec1d0944|Submitted1concurrentrangerequestscovering1ranges|172.ip.ip.ip|96|SharedPool-Worker-2bedd77a0-159e-11e6-af1d-5b2aec1d0944|beded730-159e-11e6-af1d-5b2aec1d0944|Executingseqscanacross3sstablesfor[min(-1),min(-1)]|172.ip.ip.ip|382|SharedPool-Worker-4bedd77a0-159e-11e6-af1d-5b2aec1d0944|bedefe40-159e-11e6-af1d-5b2aec1d0944|Read7liveand0tombstonecells|172.ip.ip.ip|2057|SharedPool-Worker-4bedd77a0-159e-11e6-af1d-5b2aec1d0944|bedf2550-159e-11e6-af1d-5b2aec1d0944|Read2liveand0tombstonecells|172.ip.ip.ip|2495|SharedPool-Worker-4bedd77a0-159e-11e6-af1d-5b2aec1d0944|bedf7370-159e-11e6-af1d-5b2aec1d0944|Read1liveand0tombstonecells|172.ip.ip.ip|3066|SharedPool-Worker-4bedd77a0-159e-11e6-af1d-5b2aec1d0944|bee00fb0-159e-11e6-af1d-5b2aec1d0944|Read17liveand32tombstonecells|172.ip.ip.ip|16892|SharedPool-Worker-4bedd77a0-159e-11e6-af1d-5b2aec1d0944|bee05dd0-159e-11e6-af1d-5b2aec1d0944|Read7liveand0tombstonecells|172.ip.ip.ip|18757|SharedPool-Worker-4bedd77a0-159e-11e6-af1d-5b2aec1d0944|bee084e0-159e-11e6-af1d-5b2aec1d0944|Scanned5rowsandmatched5|172.ip.ip.ip|19172|SharedPool-Worker-4The results fromcassandra-tracingcan be limited by thresholds on both duration and tombstones, and by and a result cap. The result cap gathers the same number of results but caps the display.  When limited results are displayed the results displayed are always the longest duration events.Regardless of limiting mechanisms, note that full table scans must be done on thesessionsandeventstables in thesystem_traceskeyspace.Configurations are set by command line flags, these can be accessed by runningcassandra-tracing –help:node_ip:Specify ip of the node being queried–tombstoneThreshold int:Show only sessions who read tombstones equal to or greater than this threshold.  Default 0.–time int:Show only sessions that took longer than this threshold in microseconds.  Default 10000 (10ms).–resultCap int:Maximum number of results to print out. To print all results, use 0.  Default 100.–slim:Enable slim mode, where the query and time started are suppressed in the output.Supported Python VersionsPython 2.7 and Python 3.4+ are supported.LicensesThe cassandra-toolbox pacakge is licensed under the MIT license.IssuesPlease report any bugs or requests that you have using the GitHub issue tracker!AuthorsDr. Jeffrey BergerDr. Joshua WickmanCarlos Monroy Nieblas