Gamua/Starling-Framework — pull request #638 by stepan-beresnev (2014-11-04 13:20:55+00:00)
Title: Optimization: double vertex and index buffers usage during Stage.drawToBitmapData

In Facebook web games to show html (FB popups, for example) over flash with wmode direct we are doing stage screenshots, jpeg them, encode into base64 string and pass it to javascript, which in turn hides flash and replaces it with image received. I think, that’s usual approach for Starling Facebook games.
The problem is that some of our users are getting errors like that sometimes:
ArgumentError: Error #3672: Buffer creation failed. Internal error.
at flash.display3D::Context3D/createVertexBuffer()
at starling.display::QuadBatch/createBuffers()
at starling.display::QuadBatch/syncBuffers()
at starling.display::QuadBatch/renderCustom()
...
which are thrown from random QuadBatches. And for some reason before this change quite a lot of such errors were thrown from drawToBitmapData function. It means, that rendering into bitmap works much worse than usual render for some reason, taking into account, that it happens let’s say once per 10 minutes, and render happens every frame. Probably that’s because buffers are reused in usual rendering.
In this change I reused Starling’s render support object.
Documentation states: “Buffer Creation Failed: if the VertexBuffer3D object could not be created by the rendering context (but additional information about the reason is not available).” So it’s not completely clear for me, why this error happens, but after this change it’s very likely, that new vertex and index buffers won’t be created at all during stage drawing into bitmap. It means twice less GPU memory usage for vertex and index buffers and speed up of the function as there is no need to create them.
Also current implementation doesn’t dispose RenderSupport object created, which can cause GPU memory leaks. And it’s not needed after the change.
The change also includes context validity check, if context isn’t valid, function won’t crash, but will return empty bitmap.
Also it will try to reconfigure back buffer, if it’s needed.
The thing, I’m not sure about is using mClippedViewPort for setProjectionMatrix’s parameters instead of stage, this way it should render only visible stage part, if I’m not mistaken. It seems to me, that it’s more correct behaviour, but I’m not sure.