Pyogp/Chat Logs/Daily Meeting/20 Aug 2008

* [9:33] Enus Linden: well, this can be a brief huddle then
* [9:33] Gareth Ellison: hi
* [9:33] Gareth Ellison: would like to ramble about 2 things in particular
* [9:33] Infinity Linden: yeah.. and it seems that Tao and Lock are the heavy lifters on the PyOGP thing
* [9:33] Gareth Ellison: let me know when i may ramble
* [9:33] Enus Linden: k, lemme update current status and then we can ramble
* [9:34] Enus Linden: so pyogp has a couple of active branches of exploration
* [9:34] Enus Linden: lock's refactoring of the messaging system
* [9:34] Enus Linden: tao's exception branch
* [9:34] Infinity Linden: mmm... Gareth... is it rambling regarding PyOGP? 'cause Zero or Whump's Office Hours might be more appropriate
* [9:34] Enus Linden: and taos logging work
* [9:34] Enus Linden: for those interested in providing feedback, please review the active topics on the pyogp mailing list
* [9:35] Infinity Linden: we're trying to stay focused on the python implementation of OGP here
* [9:35] Enus Linden: i'd like to wrap these up asap so that we can tag and move on
* [9:35] Infinity Linden: of course... sometimes that doesn work
* [9:35] Infinity Linden: but that's what we're _trying_ to do
* [9:35] Gareth Ellison: infinity - it is about pyogp
* [9:35] Infinity Linden: okay... cool
* [9:35] Enus Linden: hi tao
* [9:35] Infinity Linden: Tao's here
* [9:35] Gareth Ellison: was talking about some stuff with sai on IRC
* [9:35] Infinity Linden: I'd like to get an update from him
* [9:35] Infinity Linden: and then we'll open the floor
* [9:35] Tao Takashi: Hi there
* [9:35] Infinity Linden: so
* [9:36] Tao Takashi: from me?
* [9:36] Enus Linden: tao, i was just asking folks to take any comments per active work to the list
* [9:36] Infinity Linden: hey Tao... i have to run quick today
* [9:36] Infinity Linden: so
* [9:36] Enus Linden: infinity, what's your status
* [9:36] Tao Takashi: I have no real update today, was busy with day work
* [9:36] Infinity Linden: i was just thinking we could get an update
* [9:36] Saijanai Kuhn: I gave Gareth a copy of Lock's discussion about refactoring
* [9:36] Infinity Linden: so we make sure we don't miss anything important
* [9:37] Enus Linden: gareth, you aren't on the mailing list?!
* [9:37] Infinity Linden: okay. then... my status is i too am dealing with my day job
* [9:37] Gareth Ellison: actually, i'm not - i should be
* [9:37] Infinity Linden: and am discovering that editing the OGP spec takes a lot more time than i had ever thought
* [9:37] Gareth Ellison: but put basically, my comments were that the current message system seems rather fixed and puts contraints on client applications
* [9:37] Enus Linden: (shame on you gareth :P)
* [9:37] Infinity Linden: so.. mee too... not mych of an update
* [9:37] Gareth Ellison: i.e you must use the message library's networking code
* [9:37] Saijanai Kuhn: https://lists.secondlife.com/pipermail/pyogp/2008-August/000177.html https://lists.secondlife.com/pipermail/pyogp/2008-August/000178.html
* [9:37] Enus Linden: k infinity
* [9:38] Infinity Linden: okay... i'm done... so anyone else?
* [9:38] Infinity Linden: then we open up the floor to gareth
* [9:38] Enus Linden: tao, did you want to comment on status?
* [9:38] Gareth Ellison: ok, right now the plan i've been told is to enable eventlet or twisted as the networking portion
* [9:38] Gareth Ellison: now, what i'd like to see is the ability to do it all offline, and use one's own serialiser or to serialise to bytes offline
* [9:39] Gareth Ellison: have a message class
* [9:39] Tao Takashi: I have no real status ;-)
* [9:39] Gareth Ellison: with a to_bytes() type method, similar to libsl
* [9:39] Enus Linden: that is always open to other implementations, but that is what's been explored to date gareth
* [9:39] Infinity Linden: okay.. gareth... you officially have the floor
* [9:39] Tao Takashi: no, the goal is actually not to bind it to any networking lib
* [9:39] Infinity Linden: also.. nothing is "carved in stone"
* [9:39] Gareth Ellison: enus - the issue is that the networking portion should be up to the application, and it should also be thread safe
* [9:39] Infinity Linden: and the objective has always been to allow flexible networking
* [9:39] Tao Takashi: I was just playing around with eventlet and twisted to find the requirements for these
* [9:40] Gareth Ellison: as message state is locked to the message system rather than single messages, it's absolutely not thread safe
* [9:40] Infinity Linden: hmm... are you sure it's not thread safe?
* [9:40] Tao Takashi: that was my issue as well and I we will get rid of that in the refactoring
* [9:40] Enus Linden: i'm not clear on the details of why this is....
* [9:40] Gareth Ellison: the circuit management etc should be an optional portion, so that it can be reimplemented without modifying the library itself
* [9:40] Gareth Ellison: infinity - to build a message you use 1 message system object
* [9:40] Infinity Linden: it's not multi-threaded, but i thought eventlet could handle being enclosed in a multi-threaded environment
* [9:40] Gareth Ellison: and you tell it to create a message and add fields
* [9:41] Tao Takashi: so in the new version the messages should be independant of the circuits and the message system
* [9:41] Gareth Ellison: you can't create >1 message at any one time
* [9:41] Infinity Linden: does the application crash when you try?
* [9:41] Tao Takashi: yep, you'd need to lock it
* [9:41] Gareth Ellison: this is like the C++ llmessage module, and seems to be an almost direct port of the API
* [9:41] Enus Linden: gotcha
* [9:41] Gareth Ellison: infinity - it may crash, or it may just have weird state
* [9:41] Gareth Ellison: i doubt it'd crash totally, just throw exceptions
* [9:42] Tao Takashi: this is part of what I discussed with Locklainn and we are going to change this
* [9:42] Gareth Ellison: but still not good behaviour
* [9:42] Tao Takashi: it might simply work not as you expected
* [9:42] Gareth Ellison: follow the tao!
* [9:42] Enus Linden: are those issues raised on the list tao?
* [9:42] Infinity Linden: right... if it crashes after you put mutexes around it, then that's hardcore bad
* [9:42] Tao Takashi: if e.g. the wrong message is sent to the wrong recipient
* [9:42] Infinity Linden: (talking about eventlet)
* [9:42] Gareth Ellison: infinity - why do we need mutexes?
* [9:42] Tao Takashi: this might lead to bugs which are hard to trace actually
* [9:42] Gareth Ellison: why not just build messages in a message object? that way it doesn't matter if we have 10 billion threads building messages at once
* [9:43] Tao Takashi: so in the new design you first create a message
* [9:43] Infinity Linden: to protect single data structures from improper access from two threads
* [9:43] Infinity Linden: oh oh
* [9:43] Tao Takashi: then you can call a serializer on it which gives you bytes
* [9:43] Infinity Linden: i see where you're going with this
* [9:43] Infinity Linden: actually... i agree with you on that one
* [9:43] Enus Linden: updated proposal:
* [9:43] Enus Linden: message_system= MessageSystem()
* conn = Connection(message_system, (address, port))
* msg = Message(...)
* conn.send(msg)
* [9:43] Infinity Linden: but at the time we were building the system, it was one of the early decisions
* [9:43] Tao Takashi: but you also need a connection to the region which takes care of packet ids and acks and such
* [9:44] Enus Linden: or alternately:
* [9:44] Enus Linden: message_system= MessageSystem ()
* msg = Message(...)
* message_system.send(msg, (address, port))
* [9:44] Enus Linden: those are the current discussion points raised in Locklainn's last email
* [9:44] Gareth Ellison: tao - you can have a circuit object seperately
* [9:44] Infinity Linden: yeah... i was wanting to create "context" with respect to agent domain proxies
* [9:44] Gareth Ellison: which accepts a message object
* [9:44] Tao Takashi: so the connectin or circuit can then put the message into a packet, add an id and maybe give it back to you. This can then be serialized to bytes as well
* [9:44] Infinity Linden: but coordinating the design got to be a bit much
* [9:44] Gareth Ellison: MyCircuit = RegionCircuit((addr,port))
* [9:45] Tao Takashi: isn't that what I am saying? :)
* [9:45] Gareth Ellison: MyMsg = Message('AgentUpdate')
* [9:45] Tao Takashi: I just might call it a connection
* [9:45] Gareth Ellison: # do stuff here
* [9:45] Infinity Linden: sure
* [9:45] Infinity Linden: i think we're all on the same page
* [9:45] Gareth Ellison: MyCircuit.SEndMsg(MyMsg)
* [9:45] Enus Linden: that is totally the design being proposed
* [9:45] Gareth Ellison: do it like that and it's totally flexible
* [9:45] Tao Takashi: yep
* [9:45] Infinity Linden: just a question of when it'll be implemented
* [9:45] Gareth Ellison: yay
* [9:45] Tao Takashi: the only question is how what send_message() is doing
* [9:45] Gareth Ellison: my ramble is over then
* [9:45] Tao Takashi: and how it knows how to send a message
* [9:45] Infinity Linden: My Smalltalk language bindings for OGP do that
* [9:45] Tao Takashi: it should not have the socket
* [9:46] Tao Takashi: one way would be that it returns bytes and the address
* [9:46] Tao Takashi: and you take care of it
* [9:46] Tao Takashi: or you pass it some endpoint object which it can use
* [9:46] Gareth Ellison: send_message() should know where to send it, since it'll have the socket in the circuit object
* [9:46] Gareth Ellison: circuit.send_message()
* [9:46] Saijanai Kuhn: By way of intro, Gareth is owrking on a C++ versoin of OpenSIm and did the python proxy to link unmodded OPenSIms to the AD
* [9:46] Gareth Ellison: it uses self.remote_sock
* [9:46] Tao Takashi: having the socket in the circuit object does not work wirth twisted
* [9:46] Infinity Linden: yeah... we probably don't even need to worry about whatever the Python equivalent of a socket factory is
* [9:46] Infinity Linden: as URLLIB seems to handle https as well as http
* [9:46] Gareth Ellison: sai - not quite, i've got a C++ binding to the viewer's LLMessage for embedded python
* [9:46] Tao Takashi: you need to use the twisted functionality then
* [9:47] Gareth Ellison: and coding a sim in python
* [9:47] Enus Linden: fancy pants
* [9:47] Ash Venkman: ooh, smalltalk OGP? Interesting.
* [9:47] Infinity Linden: well... good luck...
* [9:47] Infinity Linden: I think Tao's doing the same thing
* [9:47] Tao Takashi: so that's why I played around with some more abstract endpoint object which you give to the circuit object
* [9:47] Gareth Ellison: tao's binding the viewer's llmessage module?
* [9:47] Infinity Linden: so maybe you two should definitely be coordinating
* [9:48] Tao Takashi: and this might be a shared socket, a separate socket per circuit or the some code which use twisted
* [9:48] Ash Venkman: by the way, i'm a Twisted developer, so if you have any questions or issues with it, i'd be interested in hearing about them
* [9:48] Tao Takashi: I never looked at the viewers module ;-)
* [9:48] Tao Takashi: ash: very good to know!
* [9:48] Enus Linden: yay!
* [9:48] Tao Takashi: are you on the list, Ash?
* [9:48] Infinity Linden: Gareth..
* [9:48] Infinity Linden: Ash
* [9:48] Tao Takashi: or in the channel?
* [9:48] Infinity Linden: https://lists.secondlife.com/cgi-bin/mailman/listinfo/pyogp
* [9:48] Ash Venkman: i'm on #pyogp as 'dash'
* [9:48] Infinity Linden: is the url for the mailing list
* [9:48] Infinity Linden: and
* [9:48] Tao Takashi: cool
* [9:49] Infinity Linden: yeah.. #pyogp is on freenode?
* [9:49] Infinity Linden: is that right?
* [9:49] Ash Venkman: yes
* [9:49] Enus Linden: yep
* [9:49] Tao Takashi: so I still need to wrap my head around it and try things. I will read Locklainn's messages later and will reply to them
* [9:49] Enus Linden: thanks tao
* [9:49] Infinity Linden: cool
* [9:49] Tao Takashi: and give some example of where I see problems or where things migth be different
* [9:49] Tao Takashi: in the end I think the applicatioin definitely needs to decide how networking is done
* [9:49] Tao Takashi: I also need to revitis how TCP/HTTP is done regarding this
* [9:50] Saijanai Kuhn: especialy if we have two such drasticaly different usecases as a multi-avie test harness and a lightweight client app
* [9:50] Tao Takashi: as you also might want to be able to handle this in e.g. the twisted event loop instead of using some blocking way in threads
* [9:50] Infinity Linden: and i need to hurry up and get the spec draft finished
* [9:50] Tao Takashi: so if all does not work, the circuit maybe should at least be able to attach a packet id and check acks and give you back some bytes and an address to send it to
* [9:50] Gareth Ellison: modules, modules, modules - modular raw networking, modular circuit management, modular serialisers
* [9:51] Tao Takashi: so you can still implement your own way of sending if you need to
* [9:51] Enus Linden: that's the beauty of zca gareth
* [9:51] Tao Takashi: and we of course also need a way to give some bytes to the circuit to handle it
* [9:51] Tao Takashi: don't talk about ZCA to Gareth ;-)
* [9:51] Gareth Ellison: the circuit t akes a message
* [9:51] Infinity Linden: yeah... ZCA gives you factory patterns with registration
* [9:51] Gareth Ellison: not raw bytes
* [9:52] Tao Takashi: Gareth. But if a message comes in you have bytes
* [9:52] Gareth Ellison: don't get me on the ZCA subject, just think modules, modules
* [9:52] Gareth Ellison: tao - then you keep it wrapped in a message object
* [9:52] Gareth Ellison: which should be possible with a sane constructor
* [9:52] Enus Linden: k, that's the beauty of our modular approach then
* [9:52] Tao Takashi: but when I receive data on a socket I don't yet have a message object
* [9:53] Tao Takashi: but we can pass it to a deserializer which then creates the message object
* [9:53] Gareth Ellison: you do if do Message(data=in_pack[0])
* [9:53] Tao Takashi: so the question is maybe who does the task of deserialization
* [9:53] Tao Takashi: as the circuit seems to do serialization I wonder why it shouldn't do deserialization. Which is in any way in a separate module of course
* [9:53] Ash Venkman: accepted your inventory offer.
* [9:53] Tao Takashi: so you can exchange it if you need to
* [9:54] Tao Takashi: more likely the question is: who is going to call it, the circuit or the application
* [9:54] Gareth Ellison: yeah, first get your IDeserialiser from your IFactory object, a child of it, which implements your IDeserialiserFactroy interface and gives you an IDeserialiser implementation which.....
* [9:54] Infinity Linden: okay... i gotta run
* [9:54] Tao Takashi: cya Infinity
* [9:55] Gareth Ellison: Message class does deserialisation either within itself or using a BinaryMessageReader class
* [9:55] Gareth Ellison: it ain't rocket science ;)
* [9:55] Tao Takashi: I don't like that ;-)
* [9:55] Tao Takashi: I think the message should not know anything about how to serialize or deserialize itself
* [9:55] Tao Takashi: because then you are more flexible in changign the format
* [9:55] Tao Takashi: you can still use the same message object even if JSON comes in
* [9:56] Gareth Ellison: explain when we're going to change the format of a binary message?
* [9:56] Gareth Ellison: when we use the data= keyword param or the to_bytes() method
* [9:56] Ash Venkman: JSON? not Protocol Buffers? :)
* [9:56] Tao Takashi: but we can of course put a convenience method inside the message class which invokes the deserializer and the serializer
* [9:56] Gareth Ellison: use json= in constructor or to_json() when JSON comes along
* [9:56] Tao Takashi: I would actually like protobufs :)
* [9:57] Ash Venkman: (I just wrote some protobuf-parsing code in C, so it's fresh on my mind :)
* [9:57] Tao Takashi: more likely use some parameter which defines the output format
* [9:57] Tao Takashi: serialize(data, format="application/llsd+xml")
* [9:58] Tao Takashi: so you don't need to change the class to add a new format
* [9:58] Tao Takashi: but I think these are details
* [9:58] Tao Takashi: my main goal is to enable any networking lib to be used
* [9:58] Tao Takashi: and even use none at all
* [9:59] Tao Takashi: and as I understand that's our common goal ;-)
* [9:59] Gareth Ellison: more messy than my tastes, but that works
* [9:59] Ash Venkman: that sounds pretty strange to me.
* [9:59] Tao Takashi: what, Ash?
* [10:00] Ash Venkman: networking-lib independence
* [10:00] Tao Takashi: why? I think it should be the application which needs to decide it
* [10:00] Tao Takashi: and it seems possible
* [10:00] Ash Venkman: sure, it's possible
* [10:00] Ash Venkman: I'm just curious what benefits it brings
* [10:00] Graph Weymann: use case! use case! :)
* [10:01] Saijanai Kuhn: also, keep in mind that as a test harness, pyogp might handle 1,000 avatars simultaneously, while as a plugin for a IM app, it would be doing things entierly differently
* [10:01] Ash Venkman: Sure.
* [10:01] Tao Takashi: so one use case might be LL who wants to use eventlet to implement massive parallel tests
* [10:01] Tao Takashi: but some might want to use twisted for e.g. GUI integration
* [10:01] Gareth Ellison: wants to code sims if this thing is sane to use
* [10:01] Tao Takashi: also we maybe don't want to depend on eventlet
* [10:01] Tao Takashi: or on twisted
* [10:01] Saijanai Kuhn: or the lightweight IM client plugin
* [10:02] Ash Venkman: I guess I'm biased, I don't see any reason to not depend on twisted ;-)
* [10:02] Saijanai Kuhn: or a plugin to Maya or Blender to directly load sculpty textures
* [10:02] Ash Venkman: well
* [10:02] Tao Takashi: because twisted is strange ;-)
* [10:02] Tao Takashi: at least on first sight
* [10:02] Graph Weymann: Twisted can run on someone else's event loop right?
* [10:02] Gareth Ellison: twisted is.... twisted
* [10:03] Ash Venkman: graph, it can integrate with other event loops
* [10:03] Tao Takashi: and it might be a barrier of entry. I know experimented with it this weekend but tried many times to actually understand it
* [10:03] Saijanai Kuhn: Donovan worked on twisted and recommended eventlet for this originally
* [10:03] Tao Takashi: I am still not sure I did fully ;-)
* [10:03] Ash Venkman: it's got support for various ones
* [10:03] Ash Venkman: saijani: I know. Donovan is... interesting. :)
* [10:03] Tao Takashi: and eventlet has it's issues, too. like no UDP support in the API
* [10:03] Saijanai Kuhn: Donovan Linden = DOnovan Preston
* [10:03] Tao Takashi: and instead does lots of patching
* [10:03] Tao Takashi: which maybe can be changed
* [10:03] Ash Venkman: yeah, i worked with him before he was at LL
* [10:04] Tao Takashi: so I think these are enough reasons to keep it networking lib independant ;-)
* [10:04] Ash Venkman: yeah, it saves argument :)
* [10:04] Gareth Ellison: all you need is this:
* [10:04] Tao Takashi: we can then give examples on how to use the individual ones
* [10:04] Gareth Ellison: from socket import *
* [10:04] Ash Venkman: winces
* [10:05] Gareth Ellison: personally, i'd be happy with a bunch of struct format strings
* [10:05] Ash Venkman: that's never a good idea =/
* [10:05] Tao Takashi: this is not what I want to use ;-)
* [10:05] Tao Takashi: but Gareth should be able to use it
* [10:05] Gareth Ellison: it passes the Gareth test for possible, not for optimal
* [10:05] Tao Takashi: nobody knows what's optimal anyway ;-)
* [10:06] Tao Takashi: esp. if you look back on what you've written a week back
* [10:06] Tao Takashi: or was it an hour?
* [10:06] Gareth Ellison: gets msg_macros.h out of mind
* [10:06] Gareth Ellison: *cough*
* [10:06] Gareth Ellison: back on topic....
* [10:07] Tao Takashi: so I will read those messages later and reply to the list
* [10:08] Tao Takashi: and we also need to see how our solution then works in practice and maybe change it
* [10:08] Enus Linden: i need to run. thanks for the discussion y'all. quite useful
* [10:08] Tao Takashi: we cannot yet come up with all possible use cases anyway
* [10:08] Saijanai Kuhn: wishes we had Lock for another few weeks :-/
* [10:08] Enus Linden: indeed
* [10:08] Tao Takashi: yep
* [10:08] Tao Takashi: take care Enus!
* [10:08] Enus Linden: i need to find another dev or 2
* [10:09] Enus Linden: we won't be able to debate preferences if only tao and i are at it
* [10:09] Saijanai Kuhn: Well Gareth is opnionated ;-)
* [10:09] Enus Linden: waves
* [10:09] Gareth Ellison: i am, but a bit too
* [10:09] Gareth Ellison: i'd want to scrap ZCA totally
* [10:09] Gareth Ellison: and generally be a classhole (in the xkcd sense)
* [10:10] Saijanai Kuhn: thats something for you and Tao to take to IRC I think
* [10:12] Tao Takashi: I am not taking that anywhere ;-)
* [10:12] Saijanai Kuhn: The other usecase that makes a dedicated networking lib inadvisable is that pyogp may become the example code for the OGP itself
* [10:13] Tao Takashi: but if you want the examples to run you need some networking lib ;-)
* [10:14] Saijanai Kuhn: true, but the extreme example is the .net-etness of libsl.
* [10:14] Tao Takashi: well, that's not the networking lib
* [10:14] Ash Venkman: well, as long as the OGP APIs are all asynchronous, i'll be happy :)
* [10:14] Saijanai Kuhn: its too bound to one library to be submitted to a standards body
* [10:14] Tao Takashi: you can say, that's the python-ness for us ;-)
* [10:15] Tao Takashi: well, you submit the spec anyway.. do you need example code?
* [10:16] Tao Takashi: but actually I gotta go.. will be online later again
* [10:16] Tao Takashi: cya
* [10:17] Ash Venkman: bye
* [10:17] JayR Cela: byee everyone / byee Sai
* [10:17] Saijanai Kuhn: Bye JayR
* [10:17] Saijanai Kuhn: So I'm thinking the meeting is over transcripts 2b posted
Category: Grid Interoperability Chat Logs
Category: AW Groupies Transcripts
Category: Pyogp Transcripts