Python TCP server - CPU usage continually increasing over time - Page 1
Forum section: electronics / general

author: Delta
author_rank: Super Contributor
timestamp: January 26, 2018, 08:33:07 pm
content: Please take it easy on me, I only started learning Python last week, and have cobbled this together, so I am fully aware that it will look a pile of crap to the proper programmers on here!This is a Python3 script running on Ubuntu Server on a Raspberry Pi 3. It is emulates a Modbus TCP slave (server). This code runs in it's own thread, the main thread pulls data from an ASCII TCP stream and puts the required values into variable, which the Modbus thread then maps into a virtual register space, and responds to Modbus requests from the Master (client). The requests come in about 6 times a second, there is only one client, and every request is the same. (This whole bodge job is to get live data into a a proprietary control system that only talks Modbus TCP)It does work, and seems to work well initially. (I know I am playing with fire by assuming that the complete Modbus request is received on a single recv() call)When it starts, Python consumes around 3% of one CPU core, but this creeps up and up and up. (I am seeing this via htop, with tree view enabled. This child thread is using the CPU - if the parent shows 10% this thread show 9% etc)After about 10 hours it's at 90% and things fall apart.Whilst a total beginner with Python and socket programming, I cannot understand why this happens over time. The loop is doing exactly the same thing after the first minute than it is after 10 hours, so why the increase in CPU usage? Memory usage does not increase.Anyway, here is the monstrosity. At this point, please don't comment on the rubbish way I'm doing things, just on why the CPU usage is increasing! I'm sure it's a schoolboy error with my handling of the sockets. (As an aside, I am extremely impressed with Python, in that with nothing more than a PDF tutorial downloaded for free, Mr Google and a couple of days, I had something running that does the job, if only for a few hours!Code:class modbusThread(threading.Thread):def __init__(self):threading.Thread.__init__(self)### Now we define what will actually run as that thread - the Modbus server itselfdef run(self):global modbus_outer_loopsglobal modbus_inner_loopsglobal modbus_requests_served# create 64 kbyte address space for Modbus registers# each register is 16 bits wide, we are using 32 bit Floats, so each value takes two registers# registers thus must be addressed 0, 2, 4 etc...# any registers read that have not had WITS data placed in them will read -777.1register_address_space = bytearray(struct.pack(">f", -777.1)) * 16384wanted_record_items = [ [] ] * 100 # create empty list of lists############################## WHAT DATA ITEMS DO WE WANT? ###############################wanted_record_items = [40,41]wanted_record_items = [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]host = socket.gethostname() #MODBUS_IP_ADDRESSprint ("MODBUS- Hostname:", host)port = 502sinewave_degrees = 0.0while True: ### main outer loop. Should only run to initially bind to the TCP port.modbus_outer_loops += 1hard_restart_required = Falseprint ("MODBUS- Creating modbus_socket")modbus_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)print ("MODBUS- Binding to",host,port)try: # try to bind to the TCP port, will fail if network not up...modbus_socket.bind((host,port))except:print ("MODBUS- Cannot bind to the Modbus TCP port! Is the network up?")time.sleep(RESTART_WAIT)continuemodbus_socket.listen(0)while True: ############# Inner loop, deals with connections from clientmodbus_inner_loops += 1print ("MODBUS- Waiting for incoming connection on",host,port,"...")client_socket,addr = modbus_socket.accept()client_socket.settimeout(TCP_TIMEOUT)print ("MODBUS- ",modbus_outer_loops,", ",modbus_inner_loops,", Incoming connection from %s" % str(addr))while True:sinewave_degrees += 0.25if sinewave_degrees >= 360:sinewave_degrees = 0sinewave_float = math.sin(math.radians(sinewave_degrees)) * 50sinewave_float += 50sinewave_int = int(sinewave_float)# Read the wanted data into the register spaceregister = 0 # Register 0 gets the test signal (sine wave)register_address_space[0:3] = bytearray(struct.pack(">f", sinewave_float))register += 2 # We use TWO 16 bit registers per float!for record in list(range(100)): # loop through all records ( 0 - 99 )for item in list(wanted_record_items):register_address_space[register * 2:(register * 2)+3] = bytearray(struct.pack(">f", record_item))register += 2try:request = client_socket.recv(SOCKET_BUFFER)except:print ("MODBUS- Connection failed.")print ("MODBUS- Closing socket and restarting listener.")breakif not request:print ("MODBUS- Connection established but no data received. (Connection closed by peer?)")print ("MODBUS- Hard restarting. Killing server socket and restarting listener from scratch")hard_restart_required = Truebreakstart_time = timer()# Build the Modbus response#print ("MODBUS- -Request:"," ".join("%02x" % b for b in request))transactionID0 = requesttransactionID1 = requestprotocol0 = requestprotocol1 = requestunitID = requestfunction_code = requeststart_register_hi_byte = request start_register_lo_byte = requestnumber_of_registers_hi_byte = requestnumber_of_registers_lo_byte = requestfunction_code = 3 # function code 3 is all you get! take it or leave it!start_register = start_register_lo_byte + (start_register_hi_byte * 256)number_of_registers = number_of_registers_lo_byte + (start_register_hi_byte * 256)byte_count = number_of_registers * 2 # registers are 16 bitlength = byte_count + 3 # data length plus, byte count, func code and unit IDlength_hi_byte = length >> 8length_lo_byte = length & 255response_list = [transactionID0, transactionID1, protocol0, protocol1, length_hi_byte, length_lo_byte, unitID, function_code, byte_count] # add all that shit together as a listresponse_list += register_address_space[start_register * 2 : (start_register * 2) + (number_of_registers * 2)] # add the data onto the end of the responseresponse_bytes = bytes(response_list) # turn it into a byte string#print ("MODBUS- Response:"," ".join("%02x" % b for b in response_bytes))end_time = timer()client_socket.send(response_bytes)modbus_requests_served += 1print ("MODBUS- ",modbus_outer_loops,", ",modbus_inner_loops," - ",modbus_requests_served," requests served. Response time = ", end_time - start_time, sep="")time.sleep(0.025)print ("MODBUS- EXITING INNER LOOP-------------------------------")client_socket.close()if hard_restart_required:hard_restart_required = Falsebreaktime.sleep(MODBUS_RESTART_WAIT)print ("MODBUS- EXITING OUTER LOOP---------------------")modbus_socket.close()The sinewave stuff is just as a test signal.I have moved the block that moves the data into the register space to before the recv() call to see if that helps. It was done after the recv(), and takes no more than 0.01 seconds to execute.Many thanks, and I promise I will do a much better job with v2, but why oh why the steady increase in CPU usage?

author: Delta
author_rank: Super Contributor
timestamp: January 27, 2018, 07:29:49 am
content: I have done some more research, and I now think that the problem lies not with the sockets, but with sharing a global variable between the parent and child threads.Using strace -c -fp PID, I see that the calls to futex (and the errors) increase as time goes on. I understand this is due to spinlocks as each thread waits to access the global variable (or list to be precise). I understand that it is bad practice to use global variables in this way, but my question remains: why does this get worse over time?

author: mrflibble
author_rank: Super Contributor
timestamp: January 31, 2018, 03:02:57 pm
content: Okay, 215 views and no replies. I read it a couple of days ago and didn't really notice one single item that would be the cause. So I thought I'd leave it to other people who maybe noticed something. Well, so much for that plan...When I said I didn't really notice one single thing that would be the cause I lied. I did see one single item that's a bit of a red flag (for me) in python:class modbusThread(threading.Thread):Python threading combined with the behaviour you describe makes me think of spinlocks slowly accumulating somewhere. Main advice based on past experience is that, if at all possible, get rid of threads in python. If it's not at all possible, make it possible, and then get rid of those threads anyways. No doubt it can be done, but more trouble than it's worth IMO.For things like firing up a daemon in the background I tend to use a simple shellscript that takes care of that. So you'd have two separate processes, two separate python interpreters running. Then you have your two processes communicate through a tcp socket or fifo or whatever you deem fit. IMO that reduces the chances of having to debug"interesting things".Random tips: Use a decent IDE, that can help you get up to speed faster. I've been using PyCharm for a while now and I really like it. Free version available here:https://www.jetbrains.com/pycharm/download/When getting weird cpu usage issues, a profiler can definitely help:https://docs.python.org/3/library/profile.htmlWhen you never modify a list, consider using a tuple. Cheaperbetterfaster, and it reduces the chance of an unintended oh-woops-modification.When only ever using a list in a for loop, consider using a generator.Ditto for range(). Quite often using xrange() instead of range() will do the trick.When creating a huge empty list of lists, and then only populating a few entries, consider using dictionaries {} as alternative. Choice would depend on your access pattern.Oh, and get a decent IDE. Did I mention that yet? It will give hints a la "use style such and such here". Or in other words, free help with coding style, which in turn may increase likelihood of code reviews.What I like about pycharm is, it does all that .... and best of all, this behaviour can be turned off. Sensible default settings, but if you want popup such-and-such to go away, you can actuallyfindthe configuration option, and then turn it off.Anyways, good luck with your modbus script.

author: Delta
author_rank: Super Contributor
timestamp: February 01, 2018, 11:26:08 am
content: Thank you for your detailed reply Mr F!I have since split the script into an ASCII thread and Modbus thread, all the main script does is start the two worker threads. I have also added a lock for the big global list, and an event so ASCII thread to signal to the Modbus thread when new data is available. This keeps CPU usage below 1%, but has dramatically reduced the rate at which Modbus responses are served.Thanks for the advice of avoiding threading in Python if possible. I will look into rewriting it all and using non-blocking sockets so the ASCII receiving and Modbus serving can be done within the same loop.

author: imidis
author_rank: Frequent Contributor
timestamp: February 01, 2018, 12:33:10 pm
content: I'm not familiar enough with python. But it reminds me of a couple decades ago running a linux email server at home. It would be fine for a while then would get to a point it would be completely unresponsive. One configuration flag in sendmail was not set and causing an endless loop with itself up to the point it would take down the server.

author: paulca
author_rank: Super Contributor
timestamp: February 01, 2018, 03:17:19 pm
content: You are creating listening sockets in a loop and then inside a nested loop accepting connections?That isn't how to handle client connections. You will only be able to handle one connection at a time.You have so many nested loops allocating memory and looping over arrays that it's possible you are appending to one of them accidentally. Does memory foot print also increase?I would divide the code up into smaller parts.Try this, it will make things a LOT easier for the sockets.https://docs.python.org/2/library/socketserver.htmlI'm in work, I can post my code for handling TCP clients and UDP updates later. It also uses a global variable or two.As a side question, if you are just reading ModBus and sending it to a single end point, why not just send a UDP packet?

author: djacobow
author_rank: Super Contributor
timestamp: February 01, 2018, 04:36:20 pm
content: Multi-threaded programming has never really been Python's strongest suit. Most Python programmers who need concurrency choose twisted, gevent, or now, the one with the most support and built into Python, asyncio. Under asyncio, you only have one thread, but essentially all your code becomes nonblocking and there is an event loop that calls all the "threads". You should look at it. A drawback for me for asyncio is that it is kind of an all-or-nothing choice. You can start a program and then add some async behavior with asyncio. You sort of have to dive into the deep end.If you are going to stick with threads, then let me suggest a very simple model where the threads communicate through queues and ONLY through queues. Python even has the Queue module for this. Particular for a master / worker(s) arrangement, this works nicely with little confusion. The master can create some work, and push it into a queue and move on and do whatever (answer the next tcp request, for example) and the workers just pop from the Queue and work on it and when they are done they can push the answer into a results queue, which the master thread can check whenever.You got some advice to use an IDE. Most people prefer them, but I do not. I use a standalone text editor with syntax highlighting. It also had features for syntax checking which I leave on, for PEP-8 compliance (which I leave OFF), and even code completion (which I generally find not that useful, but others swear by it).Since you're using python3, you are already using "xrange" (python2's "xrange" is python3's "range", and the old range is gone)Best,Dave J

author: paulca
author_rank: Super Contributor
timestamp: February 01, 2018, 04:52:29 pm
content: Python threads are apparently serialized anyway. There is comment in the documentation stating that due to the way they do threading only one thread will be executing at a time.As to whether a thread can be interrupted while carrying out a non-atomic operation is another question entirely.That is not to say you can't do proper multi-processor parallel execution of threads in Python, I'm sure you can, but I don't think the default out of the box threading lib supports it.The OPs code, as far as I can determine could just delete the threading and put the run(self) into a main method. It would run the same. It only supports one client at a time and thus does not require threads, except to free the main method for other executions while waiting on socket.accept()On NBIO. It's really the empires new clothes. Behind the curtain of blocking IO the OS is using non-blocking "select" polling and it's own events model, then sending signals to release your code from it wait state. Below that layer the hardware is using signals and wait states. Sleep, wake on interrupt.It does give you control and avoids using threading libraries, but the under lying mechanisms are much the same.The other very obvious downside of NBIO is it usually requires polling. Polling is much more expensive, especially on a concurrent system with competing threads. you either poll slowly which gives your process poor performance or you poll fast (spin lock style) which gives other processes a hard time.You don't get anything for free.That said there is a tendency for people to use too many threads badly and end up in a complicated mess. A "Threading model" is a very important design decision to make early in a project.For the OPs purposes, the basic Python SocketServer lib is all he needs and multi-threading is absolutely fine. I don't think there will be much if any conflict with the shared variables, even if he supports multiple clients. At the very worse case he will get a dirty read on a multi-byte write from the ModBus thread.

author: andersm
author_rank: Super Contributor
timestamp: February 01, 2018, 05:39:24 pm
content: Quote from: paulca on February 01, 2018, 04:52:29 pmThat depends on the implementation. In CPython ("normal" Python) only one thread can be executing in the interpreter, but if it blocks eg. on a system call then another thread can be picked for execution.

author: Delta
author_rank: Super Contributor
timestamp: February 15, 2018, 10:52:28 pm
content: Quote from: paulca on February 01, 2018, 03:17:19 pmThis is a very specific application, it is not a Server in the conventional sense. There will only ever be one connection, and always from the same client. The loops are just to ensure that if the connection is closed or dropped, my script is ready to accept a new connection ASAP.I am not reading Modbus, I am reading an ASCII stream. The problematic portion of the script is the part thatServesto Modbus responses as the requests come in from the one and only client. (Which will not talk UDP)(I will look into that library though, thanks for the link)

author: Delta
author_rank: Super Contributor
timestamp: February 15, 2018, 11:01:14 pm
content: I have made a couple of improvements, such as using a queue.Queue to pass data from one thread to the other, so there are now no global variables or locks at all.I have also found the exact portion of the code that is taking longer and longer to execute as time goes on:Code:start_time = timer()register = 2for record in list(range(100)): # loop through all records ( 0 - 99 )for item in list(wanted_record_items):register_address_space[register * 2:(register * 2)+3] = bytearray(struct.pack(">f", record_item ))register += 2pass # end of loop what reads from WITS listmodbus_wits_reads += 1read_time = ( timer() - start_time ) * 1000When the script has been running for only a few minutes, this block takes around 1ms to execute.It has now been running for 3 hours and 24 minutes, this block has been executed 9948 times, and is now takingover 8msto execute!Please please PLEASE can someone point out the stupid error I have made here?EDIT-------Here is the complete code for the thread, if that makes it any clearer (I know people not posting full context is an annoyance)Code:### Now we define what will actually run as that thread - the Modbus server itselfdef run(self):modbus_outer_loops = 0modbus_inner_loops = 0modbus_requests_served = 0modbus_wits_reads = 0empty_queues = 0QUEUE_GET_TIMEOUT = 0.1libc = 'libc.so.6'for cmd in (186, 224, 178):TID = ctypes.CDLL(libc).syscall(cmd)if TID != -1:print ("Modbus PID = ", TID)break# create 256 register address space for Modbus registers# each register is 16 bits wide, we are using 32 bit Floats, so each value takes two registers# registers thus must be addressed 0, 2, 4 etc...# any registers read that have not had WITS data placed in them will read -777.1register_address_space = bytearray(struct.pack(">f", -777.1)) * 256wanted_record_items = [ [] ] * 100 # create empty list of lists############################## WHAT WITS ITEMS DO WE WANT? ###############################wanted_record_items = [40,41]wanted_record_items = [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]host = socket.gethostname() #MODBUS_IP_ADDRESSprint ("MODBUS- Hostname:", host)port = 502while True: ### main outer loop. Should only run to initially bind to the TCP port.modbus_outer_loops += 1hard_restart_required = Falseprint ("MODBUS- Creating modbus_socket")modbus_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)print ("MODBUS- Binding to",host,port)try: # try to bind to the TCP port, will fail if network not up...modbus_socket.bind((host,port))except:print ("MODBUS- Cannot bind to the Modbus TCP port! Is the network up?")time.sleep(RESTART_WAIT)continuemodbus_socket.listen(0)while True: ############# Inner loop, deals with connections from client (MSI SBC)modbus_inner_loops += 1print ("MODBUS- Waiting for incoming connection on",host,port,"...")client_socket,addr = modbus_socket.accept()client_socket.settimeout(TCP_TIMEOUT)print ("MODBUS- ",modbus_outer_loops,", ",modbus_inner_loops,", Incoming connection from %s" % str(addr))################################ MAIN RESPONSE LOOP ####################################### this loop reads from the WITS listwhile True:# retrieve the list from the queuetry:record_item = wits_data_q.get(True, QUEUE_GET_TIMEOUT)except:pass # if there's nothing waiting in the queue, do nowt...empty_queues += 1else:# if there is, update the registers...# Read the wanted WITS data into the register space#register = 0 not used (was a test signal)#register_address_space[0:3] = bytearray(struct.pack(">f", sinewave_float))#register += 2 # We use TWO 16 bit registers per float!start_time = timer()register = 2for record in list(range(100)): # loop through all records ( 0 - 99 )for item in list(wanted_record_items):register_address_space[register * 2:(register * 2)+3] = bytearray(struct.pack(">f", record_item ))register += 2pass # end of loop what reads from WITS listmodbus_wits_reads += 1read_time = ( timer() - start_time ) * 1000try:request = client_socket.recv(SOCKET_BUFFER)except:print ("MODBUS- Connection failed.")print ("MODBUS- Closing socket and restarting listener.")breakif not request:print ("MODBUS- Connection established but no data received. (Connection closed by peer?)")print ("MODBUS- Hard restarting. Killing server socket and restarting listener from scratch")hard_restart_required = Truebreakstart_time = timer()# Build the Modbus responseprint ("MODBUS- -Request:"," ".join("%02x" % b for b in request))transactionID0 = requesttransactionID1 = requestprotocol0 = requestprotocol1 = requestunitID = requestfunction_code = requeststart_register_hi_byte = request start_register_lo_byte = requestnumber_of_registers_hi_byte = requestnumber_of_registers_lo_byte = requestfunction_code = 3 # function code 3 is all you get! take it or leave it!start_register = start_register_lo_byte + (start_register_hi_byte * 256)number_of_registers = number_of_registers_lo_byte + (start_register_hi_byte * 256)byte_count = number_of_registers * 2 # registers are 16 bitlength = byte_count + 3 # data length plus, byte count, func code and unit IDlength_hi_byte = length >> 8length_lo_byte = length & 255response_list = [transactionID0, transactionID1, protocol0, protocol1, length_hi_byte, length_lo_byte, unitID, function_code, byte_count] # add all that shit together as a listresponse_list += register_address_space[start_register * 2 : (start_register * 2) + (number_of_registers * 2)] # add the data onto the end of the responseresponse_bytes = bytes(response_list) # turn it into a byte stringprint ("MODBUS- Response:"," ".join("%02x" % b for b in response_bytes))client_socket.send(response_bytes)response_time = ( timer() - start_time ) * 1000modbus_requests_served += 1print ("MODBUS- ",modbus_outer_loops,", ",modbus_inner_loops," - ",modbus_requests_served," requests served. ", modbus_wits_reads," WITS reads. ", empty_queues, " empty queues. Read time = %.3f" % read_time, " Response time = %.3f" % response_time, " PID = ", TID, sep="")################# END OF MAIN MODBUS RESPONSE LOOP #####################################time.sleep(0.025)print ("MODBUS- EXITING INNER LOOP-------------------------------")client_socket.close()if hard_restart_required:hard_restart_required = Falsebreaktime.sleep(MODBUS_RESTART_WAIT)print ("MODBUS- EXITING OUTER LOOP---------------------")modbus_socket.close()################################################################################ END OF MODBUS SERVER##############################################################################I am going to put the WHAT ITEMS DO WE WANT bit inside the response loop, in case that is somehow growing with each iteration...

author: Muxr
author_rank: Super Contributor
timestamp: February 15, 2018, 11:19:26 pm
content: Where is:Code:record_itemComing from? First time I see a reference ofrecord_itemin your code.

author: Delta
author_rank: Super Contributor
timestamp: February 15, 2018, 11:24:45 pm
content: Quote from: Muxr on February 15, 2018, 11:19:26 pmSorry, I have posted the whole code now.(it is read from a queue, the other thread .put()s it there...)Code:################################ MAIN RESPONSE LOOP ####################################### this loop reads from the WITS listwhile True:# retrieve the list from the queuetry:record_item = wits_data_q.get(True, QUEUE_GET_TIMEOUT)except:pass # if there's nothing waiting in the queue, do nowt...empty_queues += 1else:# if there is, update the registers...

author: Muxr
author_rank: Super Contributor
timestamp: February 15, 2018, 11:38:02 pm
content: It is still unclear to me what data this carries:record_itemthis could be increasing somehow.perhaps add a debugging line and do:print(len(bytearray(struct.pack(">f", record_item ))))Because I think you're always expecting it to be 3 and that may not be the case.Also if I may have another comment. It is never a good idea to have catch all blind excepts.except:Your except should be specific to the error you're handling for exampleexcept TypeError:or whatever error you're trying to handle. You might be getting other exceptions that you might be completely blind to, because your code excepts them all blindly.edit: in your case for that queue try/expect it looks like you're waiting on Empty exception. So your expect should be.expect Queue.Empty:

author: Delta
author_rank: Super Contributor
timestamp: February 16, 2018, 12:43:10 am
content: Here is the thread that produces the data (record_item). Please forgive the shitness.I will also post the part that actually writes to this list below.Code:def run(self):WITS_IP_ADDRESS = "192.168.31.97" # IP address of Serial ServerWITS_TCP_PORT = 10001TCP_TIMEOUT = 10SOCKET_BUFFER = 1024MODBUS_RESTART_WAIT = 0RESTART_WAIT = 3START_DELIMITER = "&&"END_DELIMITER = "!!"MAX_BUFFER_SIZE = 65000MIN_BUFFER_SIZE = 1000 # not really the minimum as such, but how much is kept when it's emptiedWITS_frames_received = 0exceptions = 0WITS_outer_loops = 0WITS_inner_loops = 0full_queues = 0QUEUE_PUT_TIMEOUT = 0.5libc = 'libc.so.6'for cmd in (186, 224, 178):TID = ctypes.CDLL(libc).syscall(cmd)if TID != -1:print ("wits PID = ", TID)break######################### START OF MAIN LOOP ##################while True:print ("WITS- Setting all variables to -666.1 to indicate lack of incoming stream")record_item = [ [-666.1] * 100 ] * 100WITS_outer_loops +=1#create sockettry:WITS_socket.close() # try to close in case ocht left open, ken?except:print ("WITS- Socket didna need closing!")WITS_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)hostname = socket.gethostname()WITS_socket.settimeout(TCP_TIMEOUT)print ("WITS- Connecting to",WITS_IP_ADDRESS,"port",WITS_TCP_PORT)try:WITS_socket.connect((WITS_IP_ADDRESS, WITS_TCP_PORT))except:print ("WITS- Could not connect!")print ("WITS- Waiting",RESTART_WAIT,"seconds")time.sleep(RESTART_WAIT)continueprint ("WITS- Receiving a chunk of data...")input_buffer = "" # buffer is a STR typeWITS_frame = ""while True:WITS_inner_loops += 1try:chunk = WITS_socket.recv(SOCKET_BUFFER)except:print ("WITS- ",WITS_outer_loops,", ",WITS_inner_loops," - Could not receive!", sep="")time.sleep(RESTART_WAIT)breakif not chunk: # if the recv() call returns empty, GTF oot of here!print ("WITS- ",WITS_outer_loops,", ",WITS_inner_loops," - recv() returned empty!", sep="")time.sleep(RESTART_WAIT)breakinput_buffer = input_buffer + chunk.decode() # need to decode from a BYTE stream to the default STR typeinput_buffer_size = len(input_buffer)while input_buffer.find(START_DELIMITER) >= 0 and input_buffer.find(END_DELIMITER,input_buffer.find(START_DELIMITER)) >= 0: # keep checking the buffer for WITS shitstart_delimiter_position = input_buffer.find(START_DELIMITER)end_delimiter_position = input_buffer.find(END_DELIMITER,start_delimiter_position)if start_delimiter_position >= 0:#print ("WITS- found",START_DELIMITER,"at buffer position",start_delimiter_position)#print ("found",END_DELIMITER,"at buffer position",end_delimiter_position,"ken?")if end_delimiter_position >= start_delimiter_position:#print ("WITS- found",END_DELIMITER,"at buffer position",end_delimiter_position)end_delimiter_position += 2 # add 2 to the end position as it is a double dooferWITS_frames_received += 1#print ("WITS- Creating WITS frame",WITS_frames_received,"from from buffer positions",start_delimiter_position,"to",end_delimiter_position)WITS_frame = input_buffer[start_delimiter_position:end_delimiter_position]# convert frame into individual linesWITS_lines = WITS_frame.splitlines()line_num = 0# this is the loop that writes into the WITS record_item listfor item in WITS_lines:line = WITS_lines#print ("Line",str(line_num),"--",line)#extract data into variablesif line[:4].isdigit() and len(line) > 4: # check the line is at least 4 chars long and first 4 are digitsrecord_num = int(line[:2]) # get record numbertry:item_num = int(line[2:4])except ValueError:print ("WITS- --Exception! ValueError thrown extracting item number from line",line_num)exceptions += 1if is_float(line[4:]): # if the remainder of the line is a valid floatrecord_item = float(line[4:])print ("Variable for record ",record_num,", item ",item_num," has been assigned a value of ",record_item,sep="")line_num +=1if line_num > 100: # sanity checkprint ("WITS- Line number is over 100! Breaking from loop!")breakpass # this is the exit from the WITS record_item loop# send the new WITS data to the queuetry:wits_data_q.put(record_item, True, QUEUE_PUT_TIMEOUT)except queue.Full:pass # if the queue already has an item in it, just leave itfull_queues += 1input_buffer = input_buffer[end_delimiter_position + 1:] # empty buffer prior to end of frame#print ("WITS- Input buffer size is now",len(input_buffer))print ("WITS- ",WITS_outer_loops,", ",WITS_inner_loops," - ",WITS_frames_received," frames received, buffer = ",len(input_buffer), ". ", full_queues, " full queues. PID = ", TID, sep="")#print (modbus_requests_served,"Modbus requests served")### map data to Modbus registers....######################################pass # not sure where we are here?!!if len(input_buffer) > MAX_BUFFER_SIZE:input_buffer = input_buffer[MAX_BUFFER_SIZE - MIN_BUFFER_SIZE:]print ("WITS- BUFFER DITCHED!")print ("WITS- Aboot to close the WITS socket")WITS_socket.close()print ("WITS- Waiting",RESTART_WAIT,"seconds...")time.sleep(RESTART_WAIT)########################################################## End of WITS client ##########################################################################################The list is written to by:Code:# this is the loop that writes into the WITS record_item listfor item in WITS_lines:line = WITS_lines#print ("Line",str(line_num),"--",line)#extract data into variablesif line[:4].isdigit() and len(line) > 4: # check the line is at least 4 chars long and first 4 are digitsrecord_num = int(line[:2]) # get record numbertry:item_num = int(line[2:4])except ValueError:print ("WITS- --Exception! ValueError thrown extracting item number from line",line_num)exceptions += 1if is_float(line[4:]): # if the remainder of the line is a valid floatrecord_item = float(line[4:])print ("Variable for record ",record_num,", item ",item_num," has been assigned a value of ",record_item,sep="")line_num +=1if line_num > 100: # sanity checkprint ("WITS- Line number is over 100! Breaking from loop!")breakpass # this is the exit from the WITS record_item loop# send the new WITS data to the queuetry:wits_data_q.put(record_item, True, QUEUE_PUT_TIMEOUT)except queue.Full:pass # if the queue already has an item in it, just leave itfull_queues += 1Good shout about the exceptions - I will add the queue.Empty to it.I will also check the length of the record_item list that is getting passed to the Modbus server thread....Thanks for your help mate.

author: Delta
author_rank: Super Contributor
timestamp: February 16, 2018, 01:24:12 am
content: Right, I have addedCode:pass # this is the exit from the WITS record_item looplength_record_item = sys.getsizeof(record_item)# send the new WITS data to the queuetry:wits_data_q.put(record_item, True, QUEUE_PUT_TIMEOUT)To the producer thread to see the size of theCode:record_itemobject that is placed into the queue.AndCode:register_address_space[register * 2:(register * 2)+3] = bytearray(struct.pack(">f", record_item ))size_of_item = len(bytearray(struct.pack(">f",record_item)))if size_of_item > largest_item:largest_item = size_of_itemregister += 2to the thread that reads each item from this object.length_record_item is always 436.The largest item size is 4 bytes, just as expected, yet the execution time is increasing before my very eyes...

author: Muxr
author_rank: Super Contributor
timestamp: February 16, 2018, 02:25:52 am
content: Still analyzing the code.. (it's not an easiest structure to follow, I am sure you realize).. but right of the bat something sticks out.You might have python'spassstatement confused.passin python is no-op. It doesn't do anything. It's really only useful for for closing empty blocks. Since Python's blocks can't really be empty, for instance having an empty function doesn't work because the spacing is recognized as a begining of the block.Code:def some_function():## above would throw an errorwhich is wherepasscomes inCode:def some_function():passYou might be logically confusing it forbreakwhich can exit an existing loop.

author: Muxr
author_rank: Super Contributor
timestamp: February 16, 2018, 02:55:31 am
content: I think your slice math might be wrong:Code:[register * 2:(register * 2)+3]This doesn't look right to me.. again not entirely sure of your intentions. But I can tell you, this is not the reason why your code is running slower. I just ran this in interactive iPython:Code:# initialized wanted_record_items like you have itIn : wanted_record_items = [ [] ] * 100In : wanted_record_items = [40,41]In : wanted_record_items = [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]# and then just had your same loop print the line in the loop the code should be iterating over with appropriate indices (slice math)In : for record in list(range(100)):...: for item in list(wanted_record_items):...: print("reguster_address_space[{}:{}] = bytearray(struct.pack(\">f\", record_item[{}][{}]))...: ".format(register * 2, (register * 2) + 3, record, item))...: register += 2# result:reguster_address_space[4:7] = bytearray(struct.pack(">f", record_item))reguster_address_space[8:11] = bytearray(struct.pack(">f", record_item))reguster_address_space[12:15] = bytearray(struct.pack(">f", record_item))reguster_address_space[16:19] = bytearray(struct.pack(">f", record_item))reguster_address_space[20:23] = bytearray(struct.pack(">f", record_item))reguster_address_space[24:27] = bytearray(struct.pack(">f", record_item))reguster_address_space[28:31] = bytearray(struct.pack(">f", record_item))reguster_address_space[32:35] = bytearray(struct.pack(">f", record_item))reguster_address_space[36:39] = bytearray(struct.pack(">f", record_item))reguster_address_space[40:43] = bytearray(struct.pack(">f", record_item))reguster_address_space[44:47] = bytearray(struct.pack(">f", record_item))reguster_address_space[48:51] = bytearray(struct.pack(">f", record_item))reguster_address_space[52:55] = bytearray(struct.pack(">f", record_item))reguster_address_space[56:59] = bytearray(struct.pack(">f", record_item))reguster_address_space[60:63] = bytearray(struct.pack(">f", record_item))reguster_address_space[64:67] = bytearray(struct.pack(">f", record_item))reguster_address_space[68:71] = bytearray(struct.pack(">f", record_item))reguster_address_space[72:75] = bytearray(struct.pack(">f", record_item))reguster_address_space[76:79] = bytearray(struct.pack(">f", record_item))reguster_address_space[80:83] = bytearray(struct.pack(">f", record_item))reguster_address_space[84:87] = bytearray(struct.pack(">f", record_item))reguster_address_space[88:91] = bytearray(struct.pack(">f", record_item))reguster_address_space[92:95] = bytearray(struct.pack(">f", record_item))reguster_address_space[96:99] = bytearray(struct.pack(">f", record_item))reguster_address_space[100:103] = bytearray(struct.pack(">f", record_item))reguster_address_space[104:107] = bytearray(struct.pack(">f", record_item))Does that look right?It ran instantly on my machine.. so the reason for the slowdown is something else. I suspect the way your server (it really looks like a client to me) code is setup.edit: excuse the typo in the list's name.. but you get the point.. there is nothing functionally wrong with that loop and it shouldn't be causing slow downs in my opinion.

author: Muxr
author_rank: Super Contributor
timestamp: February 16, 2018, 02:59:13 am
content: My recommendation is to rewrite the code to not rely on so many nesting loops. This is generally really difficult to troubleshoot and it also makes it much harder to test units of your code for sanity.I would go and break every single nested loop, into its own function. This way you can test them individually and have a much better picture of what's happening.There should really be only one main loop.. your server. Everything else should be spawned from it.

author: bson
author_rank: Unknown
timestamp: February 16, 2018, 01:53:29 pm
content: Is there a reason you can't use a simple generator pattern for this, using yield? Having to use threads and a queue just makes it unnecessarily complicated - remember that python is inherently single threaded. Your time measurement isn't execution time but wall time, and there may be something else spinning on the single underlying bound thread - and so your code might not be slow, it might be starved.https://wiki.python.org/moin/Generators

author: paulca
author_rank: Super Contributor
timestamp: February 16, 2018, 02:35:06 pm
content: I promised a quick sample of using the SocketServer library which should tidy things up.A small excerpt:Code:class ProbeReceiverHandler(socketserver.BaseRequestHandler):def handle(self):data = self.request.strip()socket = self.requestprint("Recieved from {}:".format(self.client_address))data_str=data.decode("utf-8")print(data_str+"\n")temps = data_str.split(",")for temp in temps:name = temp.split(":")value = temp.split(":")latest_temps = float(value)class ProbeReceiverThread(socketserver.ThreadingMixIn, socketserver.UDPServer):passclass DisplaySenderHandler(socketserver.BaseRequestHandler):def handle(self):message = ""sep = ""for key, value in latest_temps.items():message+="{0!s}{1!s}:{2!s}".format(sep,str(key),str(value))sep=","self.request.sendall(bytes(message, "utf-8"))print("Sent to {0!s}:{1!s}".format(self.client_address, message))class DisplaySenderThread(socketserver.ThreadingMixIn, socketserver.TCPServer):passif __name__ == "__main__":HOST, P_PORT, D_PORT = "0.0.0.0", 9999, 9998probe_server = ProbeReceiverThread((HOST, P_PORT), ProbeReceiverHandler)display_server = DisplaySenderThread((HOST, D_PORT), DisplaySenderHandler)try:th = threading.Thread(target=probe_server.serve_forever)th.start()th = threading.Thread(target=display_server.serve_forever)th.start()

author: Delta
author_rank: Super Contributor
timestamp: February 16, 2018, 10:03:14 pm
content: Thanks for all the advice, I know my code is all over the place, and I will certainly take on board the tips given.However I am still tearing my hair out as to why on earth this simple block takes ever longer to run.I have stripped out the troublesome block and am running the following test script on a laptop with an i5-3380M.Code:#!/usr/bin/python3import structimport timeimport mathfrom timeit import default_timer as timerDELAY = 0.001PRINT_ITERS = 1000iterations = 0modbus_wits_reads = 0record_item = [ [-666.1] * 100 ] * 100 # dummy data# create 256 register address space for Modbus registers# each register is 16 bits wide, we are using 32 bit Floats, so each value takes two registers# registers thus must be addressed 0, 2, 4 etc...# any registers read that have not had WITS data placed in them will read -777.1register_address_space = bytearray(struct.pack(">f", -777.1)) * 256wanted_record_items = [ [] ] * 100 # create empty list of lists############################# WHAT WITS ITEMS DO WE WANT? ###############################wanted_record_items = [40,41]wanted_record_items = [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]################################ MAIN RESPONSE LOOP ######################################while True:start_time = timer()register = 2for record in list(range(100)): # loop through all records ( 0 - 99 )for item in list(wanted_record_items):register_address_space[register * 2:(register * 2)+3] = bytearray(struct.pack(">f", record_item ))register += 2pass # end of loop what reads from WITS listmodbus_wits_reads += 1iterations += 1read_time = ( timer() - start_time ) * 1000if iterations >= PRINT_ITERS:iterations = 0print (modbus_wits_reads, "loops. Read time = %.3f" % read_time)time.sleep(DELAY)After 1000 loops the block takes .345ms to execute, then after 129000 loops it is up to 8ms!(I understand that this is not an accurate measurement, as it is "wall time")output:Code:1000 loops. Read time = 0.3452000 loops. Read time = 0.3343000 loops. Read time = 0.4334000 loops. Read time = 0.4245000 loops. Read time = 0.5186000 loops. Read time = 0.5037000 loops. Read time = 0.7028000 loops. Read time = 0.5049000 loops. Read time = 0.58510000 loops. Read time = 0.59711000 loops. Read time = 0.64412000 loops. Read time = 0.82513000 loops. Read time = 0.69114000 loops. Read time = 0.92615000 loops. Read time = 1.00116000 loops. Read time = 0.82317000 loops. Read time = 0.78118000 loops. Read time = 0.86119000 loops. Read time = 0.92920000 loops. Read time = 1.08621000 loops. Read time = 0.95722000 loops. Read time = 0.94023000 loops. Read time = 0.94324000 loops. Read time = 0.98125000 loops. Read time = 1.11426000 loops. Read time = 1.07327000 loops. Read time = 1.10128000 loops. Read time = 1.20829000 loops. Read time = 1.14730000 loops. Read time = 1.04531000 loops. Read time = 1.16432000 loops. Read time = 1.29233000 loops. Read time = 1.32834000 loops. Read time = 1.27635000 loops. Read time = 1.19736000 loops. Read time = 1.45937000 loops. Read time = 1.50638000 loops. Read time = 1.46839000 loops. Read time = 1.52940000 loops. Read time = 1.52741000 loops. Read time = 1.62942000 loops. Read time = 1.58343000 loops. Read time = 1.65844000 loops. Read time = 1.68645000 loops. Read time = 1.60246000 loops. Read time = 1.66747000 loops. Read time = 1.64248000 loops. Read time = 1.72049000 loops. Read time = 1.55750000 loops. Read time = 1.80851000 loops. Read time = 1.86452000 loops. Read time = 1.70153000 loops. Read time = 1.73154000 loops. Read time = 1.73055000 loops. Read time = 1.84756000 loops. Read time = 1.99657000 loops. Read time = 1.81258000 loops. Read time = 2.08759000 loops. Read time = 2.08960000 loops. Read time = 2.10861000 loops. Read time = 2.30962000 loops. Read time = 1.98563000 loops. Read time = 2.18064000 loops. Read time = 2.07965000 loops. Read time = 2.23666000 loops. Read time = 2.22767000 loops. Read time = 2.24068000 loops. Read time = 2.13969000 loops. Read time = 2.30370000 loops. Read time = 2.42071000 loops. Read time = 2.47572000 loops. Read time = 2.45773000 loops. Read time = 2.54674000 loops. Read time = 2.55375000 loops. Read time = 2.61376000 loops. Read time = 2.63877000 loops. Read time = 2.63078000 loops. Read time = 2.71879000 loops. Read time = 2.83780000 loops. Read time = 2.72381000 loops. Read time = 2.84282000 loops. Read time = 2.80883000 loops. Read time = 2.96284000 loops. Read time = 6.68085000 loops. Read time = 3.18386000 loops. Read time = 2.96887000 loops. Read time = 3.14788000 loops. Read time = 2.88689000 loops. Read time = 3.34990000 loops. Read time = 5.48391000 loops. Read time = 3.44592000 loops. Read time = 3.43393000 loops. Read time = 3.32594000 loops. Read time = 3.69695000 loops. Read time = 3.32596000 loops. Read time = 3.33397000 loops. Read time = 3.90198000 loops. Read time = 3.86499000 loops. Read time = 3.511100000 loops. Read time = 3.585101000 loops. Read time = 3.889102000 loops. Read time = 4.086103000 loops. Read time = 4.135104000 loops. Read time = 3.977105000 loops. Read time = 4.368106000 loops. Read time = 4.422107000 loops. Read time = 4.421108000 loops. Read time = 4.402109000 loops. Read time = 5.384110000 loops. Read time = 4.726111000 loops. Read time = 4.843112000 loops. Read time = 5.086113000 loops. Read time = 5.289114000 loops. Read time = 5.483115000 loops. Read time = 7.556116000 loops. Read time = 5.696117000 loops. Read time = 5.452118000 loops. Read time = 6.211119000 loops. Read time = 5.621120000 loops. Read time = 5.948121000 loops. Read time = 5.874122000 loops. Read time = 6.266123000 loops. Read time = 6.282124000 loops. Read time = 6.714125000 loops. Read time = 8.868126000 loops. Read time = 6.873127000 loops. Read time = 6.688128000 loops. Read time = 7.426129000 loops. Read time = 8.047What on earth is going on? NOTHING changes with each loop! Is there some Python overhead increasing with each iteration? Is there some way to "reset" this every X iterations? Garbage collection?I am really confused!

author: Delta
author_rank: Super Contributor
timestamp: February 16, 2018, 10:45:34 pm
content: FOUND IT!register_address_spaceis growing!I added ar_a_s_size = sys.getsizeof(register_address_space)to my test script, so we now haveCode:while True:start_time = timer()register = 2for record in list(range(100)): # loop through all records ( 0 - 99 )for item in list(wanted_record_items):start_address = register * 2end_address = start_address + 3register_address_space[start_address:end_address] = bytearray(struct.pack(">f", record_item ))register += 2pass # end of loop what reads from WITS listr_a_s_size = sys.getsizeof(register_address_space)modbus_wits_reads += 1iterations += 1read_time = ( timer() - start_time ) * 1000if iterations >= PRINT_ITERS:iterations = 0print (modbus_wits_reads, "loops. Read time = %.3f" % read_time, "Address space size =", r_a_s_size)time.sleep(DELAY)And the output it:Code:1000 loops. Read time = 0.076 Address space size = 289432000 loops. Read time = 0.099 Address space size = 586633000 loops. Read time = 0.123 Address space size = 835214000 loops. Read time = 0.137 Address space size = 1057045000 loops. Read time = 0.159 Address space size = 1337796000 loops. Read time = 0.181 Address space size = 1693117000 loops. Read time = 0.202 Address space size = 1904738000 loops. Read time = 0.224 Address space size = 2142819000 loops. Read time = 0.247 Address space size = 24106510000 loops. Read time = 0.271 Address space size = 27119711000 loops. Read time = 0.297 Address space size = 30509512000 loops. Read time = 0.312 Address space size = 34323013000 loops. Read time = 0.332 Address space size = 34323014000 loops. Read time = 0.446 Address space size = 38613215000 loops. Read time = 0.375 Address space size = 43439716000 loops. Read time = 0.411 Address space size = 43439717000 loops. Read time = 0.420 Address space size = 48869518000 loops. Read time = 0.444 Address space size = 48869519000 loops. Read time = 0.465 Address space size = 54978020000 loops. Read time = 0.489 Address space size = 54978021000 loops. Read time = 0.511 Address space size = 54978022000 loops. Read time = 0.553 Address space size = 61850123000 loops. Read time = 0.563 Address space size = 61850124000 loops. Read time = 0.596 Address space size = 69581225000 loops. Read time = 0.806 Address space size = 69581226000 loops. Read time = 0.807 Address space size = 69581227000 loops. Read time = 0.852 Address space size = 78278728000 loops. Read time = 0.696 Address space size = 78278729000 loops. Read time = 0.722 Address space size = 78278730000 loops. Read time = 0.770 Address space size = 78278731000 loops. Read time = 0.772 Address space size = 88063432000 loops. Read time = 0.819 Address space size = 88063433000 loops. Read time = 0.841 Address space size = 880634

author: Delta
author_rank: Super Contributor
timestamp: February 16, 2018, 10:59:32 pm
content: As a workaround I am re-initialising(?) the register_address_space object(?) by putting the statementCode:register_address_space = bytearray(struct.pack(">f", -777.1)) * 256inside the loop.Code:while True:start_time = timer()register_address_space = bytearray(struct.pack(">f", -777.1)) * 256register = 2for record in list(range(100)): # loop through all records ( 0 - 99 )for item in list(wanted_record_items):start_address = register * 2end_address = start_address + 3register_address_space[start_address:end_address] = bytearray(struct.pack(">f", record_item ))register += 2pass # end of loop what reads from WITS listr_a_s_size = sys.getsizeof(register_address_space)modbus_wits_reads += 1iterations += 1read_time = ( timer() - start_time ) * 1000if iterations >= PRINT_ITERS:iterations = 0print (modbus_wits_reads, "loops. Read time = %.3f" % read_time, "Address space size =", r_a_s_size)time.sleep(DELAY)This seems to have done the trick, but why is this necessary?

author: Delta
author_rank: Super Contributor
timestamp: February 17, 2018, 03:16:34 am
content: So something weird is going on with my bytearray object.I useCode:test = bytearray(struct.pack(">f", -777.1)) * 2to create an object with space for 8 raw bytes (and fill them with two 32bit floats)and thought thatCode:test[0:3] = bytearray(struct.pack(">f", 123.456 ))would simply overwrite the first four bytes with a new 32bit float.However...Code:>>> test = bytearray(struct.pack(">f", -777.1)) * 2>>> print (test)bytearray(b'\xc4BFf\xc4BFf')>>> print (sys.getsizeof(test))65>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (test)bytearray(b'B\xf6\xe9yf\xc4BFf')>>> print (sys.getsizeof(test))72>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (test)bytearray(b'B\xf6\xe9yyf\xc4BFf')>>> print (sys.getsizeof(test))72>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (test)bytearray(b'B\xf6\xe9yyyf\xc4BFf')>>> print (sys.getsizeof(test))72>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (test)bytearray(b'B\xf6\xe9yyyyf\xc4BFf')>>> print (sys.getsizeof(test))72>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (test)bytearray(b'B\xf6\xe9yyyyyf\xc4BFf')>>> print (sys.getsizeof(test))72>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (test)bytearray(b'B\xf6\xe9yyyyyyf\xc4BFf')>>> print (sys.getsizeof(test))72>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> test[0:3] = bytearray(struct.pack(">f", 123.456 ))>>> print (sys.getsizeof(test))89>>> print (test)bytearray(b'B\xf6\xe9yyyyyyyyyyyyyyyyyyf\xc4BFf')>>>Hmmmmmmm! Have I got the wrong end of this stick here, or is there something weird going on?(I should add that the data retrieved from this "register_address_space" in my script IS correct - the data that the Modbus client receives from my script is good)

author: paulca
author_rank: Super Contributor
timestamp: February 17, 2018, 09:34:08 am
content: Python is weird, but is [ [ ] ] really a list of lists?I would be more used to [][]This kind of stuff, you are doing, is begging for a more C like language. There is no ambiguity there with memory allocation and usually if you get something wrong you get a segfault.Can you run your compiled Python in Valgrind? Valgrind or KCacheGrind will show you memory allocation, deallocation, leaks and where they are originating.

author: Delta
author_rank: Super Contributor
timestamp: February 18, 2018, 03:43:50 am
content: Quote from: Muxr on February 16, 2018, 02:55:31 amNope, you were correct; that IS the reason my code was running slower!It needs to beCode:[register * 2:(register * 2)+4]You need to slicepastthe last item you want.DoingCode:test[0:3] = bytearray(struct.pack(">f", 123.456 ))tries to put a FOUR byte float into the first THREE positions, well it does put it into the first four positions, but what was the old fourth byte gets shoved to the right and retained.My code was working because I do these writes in sequence, so the rogue bytes got overwritten.Oh well, I learned a lot about debugging, and my shitty code is now all running fine.I promise my next script will be much neater!

author: djnz
author_rank: Regular Contributor
timestamp: February 18, 2018, 10:04:04 am
content: One way to intuitively remember it is [a:b] has, in the simplest case, length b-a. This has helped me avoid off-by-one errors in a number of situations.